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Executive Summary 


The Literacy Design Collaborative (LDC) was created in 2009 to support teachers in 
implementing Common Core State Standards (CCSS) and embedding literacy skill development 
throughout content area curriculum. Engaged in the evaluation of LDC tools since June 2011, 
UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is 
the independent evaluator for LDC’s current federally funded i3 validation grant. CRESST’s 
evaluation study is using multiple data sources and a quasi-experimental design (QED) to 
examine LDC implementation and impact in two cohorts of schools in two large, urban school 
districts. 


This report presents the results on implementation of LDC in the large urban school 
district on the West Coast during the third year of the intervention, and the impact of the 
program across multiple years. The study schools serve largely Hispanic populations, with a high 
proportion of students qualifying for free and reduced lunch, and many English language 
learners. As of 2018-2019, participating schools included 11 from Cohort 1, which began 
implementation during 2016-2017, and 23 from Cohort 2, which commenced at the beginning 
of the 2017-2018 school year. Our primary impact analyses, presented in this report for the 
first time, pool teachers from both cohorts to measure their impact after participating in LDC 
for 2 consecutive years (2017-2018 for Cohort 1 and 2018-2019 for Cohort 2). 


The CRESST evaluation addresses research questions in three major areas: 
e Program Characteristics and Implementation 

e Contextual Factors and Implementation 

e Program Impacts 


The findings draw on multiple data sources and methods across the 5 years of the study. 
These include surveys of teachers, teacher leaders, and administrators; analysis of the quality of 
performance tasks called LDC modules’, which are a central manifestation of LDC practice; 


* An LDC module is a standards-embedded performance task assignment that explicitly guides students to write in 
response to reading complex and discipline-specific texts. The LDC module includes an assignment prompt and 
accompanying backwards-design instructional plan for teachers to implement in the classroom. 


participant interactions with LDC CoreTools, the electronic platform through which teachers 
access LDC professional development resources (online courses, existing LDC modules, module 
templates, and support for module development); data on teacher attendance, meeting 
lengths, and coach/teacher leader calls captured in professional learning communities (PLC) 
reflection forms; LDC administrative records capturing attendance at PLC sessions and 
professional development offerings for teacher leaders and administrators; and administrative 
data on students and teachers including class rosters, student demographics, and student 
performance on state standards-based assessments. We begin with the overall findings and 
then summarize participants’ perspectives on key LDC components, intermediate effects on 
teachers’ instructional strategies and practice, and effects on student outcomes. Detailed 
evidence with regard to key LDC activities, supports, and pedagogical impacts help to explain 
the mostly positive findings and offer implications for further strengthening LDC. 


Overall Findings 


Findings from both participant surveys and analyses of student outcomes reveal positive 
results for the LDC intervention: 


e Analysis of student outcomes provided evidence of the program’s effectiveness and 
confirmation for participants’ positive views. Quasi-experimental analyses 
demonstrated a statistically significant positive impact of LDC as practiced by middle 
school teachers with 2 years of program experience. For middle school students 
exposed to LDC instruction in English Language Arts (ELA), social studies/history, and 
science, effect sizes translated to a striking 9.4 months of additional learning 
compared to similar peers. The effect size for the average observed student (who 
received a smaller dose of LDC instruction) translated to a still very impressive 4.1 
months of additional learning compared to similar peers. A statistically significant 
positive impact was also found for Cohort 2 middle schools after just one year of 
implementation. Although in the positive direction, the effect estimate at the 
elementary school level after two years of implementation was considerably smaller 
than the middle school effect and not statistically significant. The study, therefore, 
does not provide evidence for an impact at the elementary school level. It is important 
to recognize some limitations of the elementary school analysis, including the inability 
to include lower grade teachers and their students due to the lack of state assessment 
data at those grade levels. The results are therefore not generalizable to the full 
population of elementary school implementers and their students. 


e Participants across all groups perceived a positive impact on student outcomes. A 
large majority of both teachers and administrators agreed that LDC helped improve 
student learning across multiple areas, including college and career readiness, literacy 
performance, writing, and content knowledge. The two most highly rated areas of 
impact, according to all three groups, were students’ ability to complete writing 
assignments and quality of students’ writing. 
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Across all 3 years of the study, teacher, teacher leader, and administrator participants 
reported very positive attitudes toward LDC. Teachers appreciated the opportunity to 
collaborate and share practice, generally reported that the range of in-person and 
digital supports was helpful, and agreed that participating in LDC positively impacted 
their practice, in areas such as engaging students in complex text, locating evidence of 
standards in final student work, and engaging students in understanding the 
performance task assignment and standards-rubric. Administrators generally saw LDC 
as a productive tool for meeting school instructional goals, but had some concerns 
with the intensive weekly time commitment needed to implement the program, with 
some administrators choosing to discontinue participation or dedicate less time to the 
program in favor of other priorities. 


Professional Learning Community and Teacher Collaboration 


Nearly all LDC teachers participated in LDC-oriented PLCs. The frequency with which 
PLCs met in 2018-2019, however, varied greatly across schools, with the number of 
recorded PLC meetings ranging between four and 20 times in the year, and averaging 
12.6 times. The average teacher’s individual attendance rate was 78%, but again there 
was great variation. About two thirds of teachers met the teacher-level fidelity 
threshold of 80% attendance. About half of schools met the program goal of three 
quarters of PLC participants attending 80% of sessions, with some schools 
experiencing challenges related to protecting planning time and ensuring that teacher 
participants attended PLC meetings. Both teacher surveys and PLC reflection forms 
indicated that PLC meetings typically lasted 45 to 59 minutes or an hour or more, and 
therefore met the LDC expectation. 


Teachers valued the collaborative nature of LDC and its PLCs. A large majority of 
teachers credited LDC with making them more likely to collaborate with other 
teachers, not only within their grade levels and content areas but outside of them as 
well. 


LDC Training and Support 


Teachers were nearly uniform in their positive attitudes about the value of their PLC 
participation. They found the PLCs a safe space for sharing instructional plans, 
problem solving, and learning to develop modules. 


Teacher leaders were almost universally reported to be highly approachable, 
supportive, knowledgeable, and helpful. 


Teacher leaders reported high satisfaction regarding the support they received from 
coaches, professional development offerings, and how the teacher leader role allowed 
them to be instructional leaders in their schools. 


Overall, LDC coaches received positive feedback on the survey, with 96% of teachers 
and 97% of teacher leaders reporting that their coaches gave them appropriate and 
timely feedback and support. Data suggest room for improvement when it comes to 
frequency and usefulness of coach feedback. While coaches were somewhat more 
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likely to meet fidelity thresholds on module comments in 2018-2019 than 2017-2018, 
peer review was used even less frequently in 2018-2019 than it was used in 
2017-2018. 


Most teachers rated CoreTools positively, which demonstrated the success of changes 
LDC made prior to the 2017-2018 school year including refinement of the content, 
sequencing, delivery of CoreTools’ instructional content, and streamlining of 
participants’ learning process. Overall, teachers were enthusiastic about much of the 
online learning content, with more than 4 out of 5 teachers rating most aspects as 
good or excellent. Almost a third of teachers, however, were concerned with the ease 
of use of the online course materials. Despite teachers’ positive attitudes, analysis of 
CoreTools data revealed that teachers were being exposed to a less online course 
content than program goals intended. There was some improvement in this area from 
2017-2018 to 2018-2019, with a small number of schools meeting fidelity of 
implementation on exposure to the instructional content, but still exposure rates were 
considerably lower than fidelity thresholds. It is possible that there wasn’t sufficient 
PLC time to cover the content with which LDC intended participants to engage. 


LDC Implementation 


Teachers typically reported adapting/creating and implementing two LDC modules 
across the school year, which meets LDC program expectations. 


Analysis of program data suggests that while nearly all participants were engaging 
with the module-building platform to some extent, the level of engagement varied 
greatly across individuals and across subgroups (role, cohort, school level, content 
area) as evidenced by the wide range in the number of views, edits, and comments 
across teachers. More in-depth analysis of the portions of the modules that teachers 
edited indicated that engagement varied greatly across teachers’ user accounts, with 
about half of user accounts associated with engagement at a basic level (editing the 
teaching task), and other accounts associated with deeper engagement (editing 
multiple portions of modules). It is important to note that in response to teachers 
collaborating on and implementing common modules in schools, LDC refined its data 
collection in the summer of 2018 to include tracking engagement data on modules 
that teachers collaborated on. The fidelity matrix and CRESST’s analyses of editing 
data were not designed to capture this shift toward a collaborative model of 
instructional design in CoreTools, and therefore may not fully capture engagement in 
the design process. 


The majority of teachers (79% to 93%) reported success in nine key areas of LDC 
module development. Teachers were most confident in selecting focus standards, 
creating the writing assignment, identifying skills needed in the module, and making 
writing assignments relevant and engaging. The module analysis, however, suggests 
that the materials adapted and created by PLC members varied in levels of completion 
and quality. 
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With regard to their classroom implementation of LDC modules, the majority of 
teachers reported success with all six key areas queried (86% to 92%). Teachers were 
most confident with engaging students in complex text, locating evidence of standards 
in final student work, and engaging students in understanding the assignment and 
rubric. 


Leadership Support 


Almost all teachers and teacher leaders reported that their administrators encouraged 
LDC participation at the school. The majority of teachers and teacher leaders agreed 
with administrators that they allocated resources to ensure that LDC teachers could 
participate in meetings. Administrators generally voiced strong support for LDC, but 
varied in their level of direct engagement with the work, with some taking a hands-on 
approach by attending many PLC meetings, and some not. 


Overall, most administrators and teacher leaders took advantage of in-person 
meetings offered by LDC. There was less consistency in terms of the frequency of 
teacher leader/coach planning calls, with about 40% of schools meeting the fidelity 
goal, and teacher leaders and coaches meeting over the phone less frequently in the 
remainder of the schools. 


Impact on Teacher Practice 


The majority of teachers reported improving their practice in seven LDC-related skills 
(79% to 88%). Teachers were a bit more likely to report impact on selecting focus 
standards for an assignment, creating standards-driven writing assignments, and 
identifying skills that students need in writing assignments (skills concentrated at the 
beginning of the LDC learning cycle). 


Over 80% of teachers agreed that participating in LDC raised their expectations for 
students’ writing, helped them incorporate writing assignments into their existing 
curriculum, and made them more likely to collaborate with other teachers on 
designing instruction. 


Among teachers who completed the survey in both 2017-2018 and 2018-2019, 
attitudes around impact on teacher skills and practices were on average even more 
positive in 2018-2019 than 2017-2018. 


Overall, CRESST’s ratings of module quality decreased from 2017-2018 to 2018-2019, 
when looking at all rated modules and those modules created by a subpopulation of 
teachers who were present in both years. This change contrasts with an increase in 
the quality of modules scored by CRESST from 2016-2017 to 2017-2018. Please note, 
however, that sample sizes are small and thus comparisons across years are 
exploratory in nature. 


Sustainability 


Survey respondents generally reported confidence that the LDC program would be 
sustained in their schools. Analysis of attrition patterns, however, reveals concerns 


about sustaining the LDC initiative in schools. Even with the substantial supports 
provided by LDC, a number of administrators decided over the course of the 3 years 
that they did not have the resources (particularly staff time) to remain in the program. 
Competition for time between different reform efforts and school priorities seemed to 
play a role in whether schools remained in the LDC program. 


Based on the exit interviews conducted by LDC staff, school leadership teams 
indicated that LDC practice was likely to continue at some level in the majority of 
schools. Some principals and assistant principals were committed to protecting 
common planning time and expanding the use of the LDC planning process to other 
grades and subjects. A handful of schools demonstrated strong commitment to LDC 
practice by purchasing LDC product licenses. In other cases, administrators were 
unlikely to provide support into the future, and therefore practice was less likely to be 
consistent and to spread across the school. 


Impact on Student Learning 


Quasi-experimental analyses revealed a statistically significant positive impact of 
middle school teachers with 2 years of LDC experience on student ELA scores. The 
dosage-dependent version of our model also suggests that students who were 
exposed to LDC instruction in a greater number of core content classes benefited 
more from the program. Supplemental analysis also indicates that impact was greatest 
for students who were exposed to LDC in half or more of their class time in core 
subjects (ELA, social studies, and science). 


The study does not provide evidence of an impact of LDC on student ELA scores at the 
elementary school level after teachers participated for 2 consecutive years. It is 
important to note that it was only possible to test the impact of LDC in upper 
elementary grades, so the finding is not generalizable to the full population of 
elementary teachers implementing LDC and their students. Also, an analysis of Cohort 
1 middle school teachers in their third year of LDC implementation did not detect an 
impact, with limited sample size due to attrition likely a factor. 


Survey respondents were nearly uniform in perceiving a range of positive impacts of 
LDC on student learning. Among teachers who completed the survey in both 
2017-2018 and 2018-2019, attitudes about LDC impact on students were on average 
even higher in 2018-2019 than in 2017-2018. 


Conclusions 


UCLA CRESST’s multiyear mixed methods evaluation of LDC as implemented in a large 
urban West Coast school district provides evidence that LDC is an effective tool for increasing 
student learning in English language arts. LDC’s theory of action predicted that teachers would 
need to participate in the program for at least 2 years to effectively deliver LDC-infused 
instruction, and positively impact student learning, as measured by student assessment scores. 
Our quasi-experimental design analyses provide confirmation for that hypothesis, particularly 
at the middle school level, with students under teachers with 2 consecutive years of LDC 


participation performing at significantly higher levels than the comparison group of matched 
students in matched schools in the same district. That effect translates to 9.4 months of 
additional learning for students exposed to LDC in all their core content area, and 4.1 months of 
additional learning for the average observed student in the study. Our dosage-dependent 
model also suggests that middle school students with higher levels of exposure to LDC (by being 
exposed to multiple teachers in different content areas implementing LDC) benefited more 
from the program. A positive statistically significant impact was also found for Cohort 2 middle 
school students after one year, which provides further evidence of the efficacy of LDC at the 
middle school level. 


A similar effect was not found at the elementary school level with the present data, 
although attrition and the inability to include lower grade elementary teachers and students in 
the analyses may have played a role. As noted above, our study also suggests that more LDC is 
better; since most elementary students were exposed to only one teacher implementing LDC, 
the program expectation of each teacher implementing two modules per year may not have led 
to elementary students receiving a sufficient dosage of LDC to affect their learning. Further 
investigation of LDC’s impact at the elementary level would be beneficial. 


Implementation data demonstrated great appreciation for LDC among teachers, teacher 
leaders, and administrators. Educators found LDC to be a helpful tool for fostering 
collaboration, creating a safe space for sharing practice, and increasing teacher skills and 
knowledge around literacy instructional design and teaching. Despite broad support, however, 
many schools and teachers did not meet fidelity thresholds related to attendance, exposure to 
instructional content, and principal observation of LDC instruction. The findings demonstrate 
the importance of school leaders and teachers understanding and committing to the program 
in advance of implementation, and school leaders dedicating substantial resources to the 
program, particularly time for common planning. Under these ideal conditions, LDC may show 
even greater impacts on teacher practice and student learning. 


In summary, our multiyear mixed methods evaluation of the LDC as implemented in a 
large urban West Coast school district provides evidence that students exposed to LDC 
instruction made significant gains in learning, particularly at the middle school level. 
Furthermore, teachers were nearly uniform in their positive attitudes about the value of their 
PLC participation. They valued the collaborative nature of LDC and its PLCs and reported 
improving their practice in LDC-related skills and improved student learning. Despite the 
challenges in implementing LDC as intended, the student outcome analyses found LDC is an 
effective tool for increasing student learning in English language arts at the middle school level. 


xi 


Literacy Design Collaborative 2018-2019 
Evaluation Report 


Jia Wang, Joan L. Herman, Scott Epstein, Seth Leon, Deborah La Torre, and Velette Bozeman 


CRESST/University of California, Los Angeles 


1.0 Introduction 


The Literacy Design Collaborative (LDC) was created in 2009 to support teachers in 
implementing Common Core State Standards (CCSS) by integrating literacy skill development 
throughout the content areas. LDC is a national community of educators providing a teacher- 
designed and research-based framework, online tools, and resources for creating both literacy- 
rich assignments and courses across content areas. Used by individual teachers, schools, and 
districts in 40 states, LDC also is a statewide-adopted strategy for Common Core 
implementation in Kentucky, Colorado, Louisiana, and Georgia. 


UCLA’s National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST), in collaboration with its partner Research for Action (RFA), engaged in the evaluation 
of LDC implementation and its impact on student learning and teacher effectiveness starting in 
June 2011, via two parallel research studies funded by the Bill and Melinda Gates Foundation. 
Those studies included an examination of LDC implementation in eighth grade social studies 
and science classrooms in Kentucky and Pennsylvania and a districtwide implementation in 
sixth grade advanced reading classes in a large district in Florida. Results for the studies are 
available in two CRESST reports (Herman et al., 2015a; Herman et al., 2015b), as well as a 
journal article published by AERA Open (Herman et al., 2016). 


Because of its earlier success in helping students and teachers in middle schools, LDC 
received an Investing in Innovation (i3) validation grant to further develop the program. The 
current i3 project focuses on developing teacher competencies through job-embedded 
professional development and the use of professional learning communities (PLCs). Teachers 
work collaboratively with coaches to develop their expertise further and design standards- 
driven, literacy-rich writing assignments within their existing curriculum across all content 
areas. Currently, CRESST serves as the independent evaluator for LDC’s federally funded i3 
validation grant. 


While the initial focus of the i3 grant was to serve teachers and students in middle 
schools, LDC responded to local conditions by expanding its implementation to also include 
elementary and high schools at both study districts: New York City Department of Education 
(NYCDOE) and a large urban school district on the West Coast (the latter of which this report is 


focused on). The comprehensive mixed-methods evaluation includes quasi-experimental design 
(QED) analyses designed to estimate the impact of LDC on student learning, and documentation 
of LDC’s impact on teacher skills and practices. Specifically, the evaluation study addresses a 
wide range of questions about program characteristics, conditions, and impacts in the context 
of the two school districts. The study measures teacher implementation and skill improvement 
with teacher surveys, analytic data from LDC’s online CoreTools module building platform, PLC 
reflection forms, and artifact analysis. While we document the core strategies of the LDC model 
as implemented and provide support for LDC improvement, the central focus of our 
comprehensive mixed-methods evaluation is to examine the impact of LDC on student learning 
using a QED. 


This report examines LDC implementation during the 2018-2019 school year in the large, 
West Coast school district, and the impact of LDC on student learning across 2017-2018 and 
2018-2019. It draws on data from two cohorts of schools, with each school housing a PLC of 
teachers who engage in professional learning about LDC and implement LDC mini-tasks and 
modules in their classrooms. The first cohort of schools began implementing LDC during the 
2016-2017 school year, and the second cohort began implementation during the 2017-2018 
school year. A parallel report focusing on implementation in NYCDOE will be prepared during 
2020 once the student outcome data become available and we finish conducting the analysis. 


In this report, we present results from (a) surveys of classroom teachers, LDC teacher 
leaders, and school administrators; (b) analyses describing how LDC participants interacted with 
the CoreTools module building platform; (c) CRESST ratings of instructional modules created by 
LDC participants; (d) analysis of the fidelity of implementation across multiple key components, 
indicators, and data sources; and (e) confirmatory and exploratory student outcome analyses 
using QED techniques. 


Survey results provide a window into how LDC was implemented in 2018-2019 (the final 
study year), the perceived utility and effectiveness of various program components, and the 
perceived impact of LDC on both teacher and student skills and knowledge. The analysis of 
CoreTools user data and modules created and adapted by teachers provides evidence on the 
level of engagement with the online platform and module design process, and the quality of the 
products created by the teachers and PLCs. Fidelity of implementation analysis, utilizing the 
fidelity matrix designed collaboratively by LDC and CRESST, provides a broad picture of how 
schools and the program as a whole performed on key fidelity indicators in 2018-2019. 


Our primary quasi-experimental analyses illuminate the impact of LDC on students, as 
practiced by teachers who amassed 2 or more years of experience with LDC. In addition, 
supplementary analyses examine the impact of Cohort 2 teachers alone and the impact on 
subgroups of students based on whether they were exposed to LDC in the prior year and how 
much exposure they had to LDC in the outcome year. 


1.1 Logic Model 


The logic model includes four key intervention components that are predicted to be the 
drivers of change in teacher practice and student learning (see Figure 1.1). These components 
include a coach-supported professional learning community formed to implement the LDC 
intervention at the school site and provide a space for teacher collaboration; asynchronous 
support from coaches primarily in the form of feedback in CoreTools through comments and 
peer review; implementation activities completed by participating teachers including module 
development and classroom implementation; and /eadership support at different levels. Note 
that the model also indicates LDC’s expectations for the level of implementation in each area. 


Figure 1.1 
LDC i3 Logic Model 
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minutes of common planning time during 
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Digital coach joins PLC every other week. 
Sequencing incorporates blocks from LDC 
online courses. 


Coach provides feedback on modules in 
CoreTools via comments and peer review 
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Teacher 
implementation 
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different levels 


Teachers develop at least one module per 
year in Years 1-2. 

Teachers teach at least two modules per year 
in Years 1-3. 


District instructional specialists visit schools 
three times per year to sit inon PLC, meet with 
principal and teacher leader, and observe 
instruction. 
Principals participate in LDC by protecting PLC 
time, setting goals for LDC work, facilitating & 
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instructional specialists, and observing LDC 
instruction. 

Teacher leaders facilitate PLC on the ground, 
collaborate with administrator, and 
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Contextual Factors: District support, school and student demographics, other policies and initiatives 


The logic model predicts that the four key components will lead to increased teacher 
expertise and skill development and more effective Common Core aligned instruction that 


incorporates ongoing formative assessment. In turn, increased teacher capacity and more 
effective instruction will lead to increased student engagement in the short term; increased 
student skill acquisition, higher test scores, and higher rates of course completion in the 
medium term; and improved college and career readiness, education attainment, graduation 
rates, and labor market outcomes in the long term. 


Note that the logic model was revised based on refinements to the program in response 
to learning from both the pilot year (2015-2016) and Cohort 1’s first year of implementation in 
2016-2017. 


1.2 Evaluation Questions 


Our evaluation questions focus on addressing three main areas: program characteristics 
and implementation, contextual factors and implementation, and program impacts. This is the 
first report to provide results on our primary outcomes analyses, estimating the impact of LDC 
on student learning, as implemented by teachers participating in the program for 2 consecutive 
years. 


1. Program Characteristics and Implementation 


a. Whoare the participating teachers and schools? Are they representative of the 
teacher/school populations of the respective district on years of teaching, 
education level, prior student performance, etc.? 


b. How is the LDC program implemented in each district? What are the core 
components (e.g., training, tools, on-site or other direct support) and who are the 
key participants? In what ways did the LDC implementation align with the 
intended model? 


c. In what ways do teachers implement the LDC tools in their classrooms? To what 
extent do teacher practices align with intended LDC practices? 


d. How are teachers utilizing the online LDC system (including online tools, 
exemplars, collaborative workspaces, and technical assistance) in terms of 
frequency and use of key features? Does this vary by teacher characteristics? What 
are teachers’ perceptions of the value and quality of the online LDC system? 


e. What types of LDC professional development opportunities are offered to and 
utilized by teachers at each school/district? Are teachers and schools satisfied with 
the LDC professional development opportunities they received? 


2. Contextual Factors and Implementation 


a. What factors facilitate or hinder successful implementation of the LDC model at 
the teacher, school, and district levels? 


b. How can implementation of the model be improved at the teacher, school, and 
district levels? 


c. 


What other educational reforms are being implemented in the participating 
schools and districts? What are their influences on LDC adoption in the schools and 
districts? Are schools able to align reform efforts? 


What are the roles of school and district leadership in shaping the LDC 
implementation? 


3. Program Impacts 


a. 


What is the impact of LDC on the academic performance of participating students 
as measured by the state assessments? 


Do the academic impacts vary by student subgroup including prior achievement, 
race, ethnicity, socioeconomic status, gender, language proficiency, and/or 
disability? Does LDC help close the achievement gap between student subgroups? 


Do the academic impacts vary by student grade level or subject? 


What is the impact of LDC on teacher skill improvement and learning as measured 
by CoreTools and by the quality of LDC modules they produce? What is the self- 
reported impact of LDC on teacher learning? 


To what extent do teachers report changes in their practice (e.g., teaching 
strategy, collaboration with others) and changes in their comfort in implementing 
CCSS during and after the LDC intervention? 


What is the relationship between the fidelity of implementation, fidelity of 
intervention, and student learning? What are the conditions and contexts under 
which LDC tool use is most effective? 


To what extent do Cohort 1 participating schools and teachers continue their LDC- 
influenced practices in the 2019-2020 school year after LDC support ends? What 
contributed to their decision to continue or stop? What factors contributed to 
their levels of continued implementation? How does Cohort 1’s actions align with 
their previously stated intentions for continuation of LDC-influenced practices as 
reported in spring 2017? To what extent do Cohort 2 participating schools and 
teachers plan to continue their LDC-influenced practices after LDC support ends? 


2.0 Study Methodology 


In this chapter, we provide an overview of the methodology behind this report on LDC 
implementation and effects. We begin by describing the various instruments and data sources 
for the analyses, including surveys of classroom teachers and teacher leaders participating in 
PLCs as well as administrators overseeing the implementation; analytic data from LDC’s 
CoreTools platform; module artifacts including samples of student work; LDC administrative 
data; and district administrative data on students and teachers used for the outcomes analyses. 
We then describe the sample of educators and schools for each of these data sources. Finally, 
we discuss the methodological approaches for the various analyses we conducted. 


2.1 Data and Instruments 


We next describe each of the data instruments and the elements they contain. Most 
variables are measured at the teacher level, which is the unit at which the LDC intervention is 
being implemented. Administrative data for the analysis of the impact of LDC on student 
learning includes school-, teacher-, and student-level variables. 


Surveys (Teachers, Teacher Leaders, and Administrators) 


This report includes findings from the third and final year of survey collection in 
2018-2019. In collaboration with LDC, CRESST previously made minor revisions to the 
2016-2017 surveys for the 2017-2018 survey administration. These refinements involved 
streamlining the language of certain questions, in particular those capturing teacher 
pedagogical practice and the perceived impacts of LDC on teachers and students. In a few 
cases, questions and items were also added to collect systematic information on program 
conditions and impacts that were observed by LDC and CRESST anecdotally. The 2018-2019 
surveys were substantively identical to the 2017-2018 surveys. 


In 2018-2019, four different surveys were administered to LDC participants playing three 
different roles: teacher, teacher leader, and administrator. Some teacher leaders were 
classroom teachers who implemented LDC with their students, while other teacher leaders 
were out-of-classroom faculty (coaches or coordinators); these two groups received different 
versions of the survey tailored to their roles. Thus, four versions of the surveys were 
administered in spring 2019: (a) teacher, (b) teacher leader (for coaches and coordinators), 

(c) teacher leader (for teachers), and (d) administrator. 


The surveys were designed to capture multiple perspectives on key aspects of LDC’s logic 
model (see Figure 1.1), and to provide data to answer the evaluation’s research questions 
presented earlier. Survey questions targeted at the three roles fall under the domains and 
subdomains in Table 2.1. Domains were selected to align with the LDC i3 logic model and with 
the CRESST evaluation questions. Note that most domains cover multiple subdomains, 
constructs, and survey questions. Professional Learning Community/Teacher Collaboration, for 
example, captures the intensity, frequency, and collaborative environment of common planning 


time; LDC Training and Support includes quality of online courses, utility and effectiveness of 


coach support, etc.; and LDC Implementation encompasses module creation, classroom 


implementation of modules, and module peer review. 


Table 2.1 


Survey Domains for Three Respondent Groups 


Teacher 
Domain Teacher leader Administrator 

LDC participation X X X 
Professional learning community/teacher collaboration X X X 
LDC training and support X X X 
LDC implementation 

e Module creation X X 

e Classroom implementation X 

e Module peer review X 
Alignment X X 
Leadership support 

e Teacher leader support X 

e School administrator support/classroom observation X X X 

e Teacher leader leadership role X X X 

e District support X X 
Impact 

e Impact on teacher practice and learning X X 

e Impact on student learning X X 
Scale-up and sustainability X X 
Facilitators and barriers X 
Areas of improvement X X X 


Teachers and administrators were asked to reflect on both LDC’s Impact on Teacher 
Practice and Learning and Impact on Student Learning. Questions within a number of domains 


further asked respondents to reflect on conditions and supports that may potentially impact 
LDC’s implementation. These domains included teachers’ perceptions of Facilitators and 


Barriers to implementation and perceptions regarding leadership roles and support for LDC at 


different levels. Teacher leaders and administrators were also asked for their perceptions 


regarding if and how LDC would be sustained and expanded within the school. Finally, all 
respondents were asked open-ended questions regarding Areas of Improvement for LDC 
implementation. Teacher, teacher leader, and administrator survey results can be found in 
Chapter 3, and the survey instruments can be found in Appendices A, B, and C. 


LDC CoreTools 


The CRESST team received the LDC program data on i3 participants’ interactions with the 
CoreTools module-building platform. The data files captured a number of behaviors including 
document page viewing, document editing, document commenting, coach use of peer review, 
exposure to LEARN instructional cycle content, and uploading of student work. The data 
contained date- and time-stamped records of participants’ activities in all of these areas. 


Using these data files, we were able to analyze variation in the number of times the 
participants viewed, edited, and commented on documents across the school year. We 
generated descriptive statistics (minimum, maximum, mean, and standard deviation) for the 
number of times participants viewed a document page, edited a module document, and 
commented on a module document. We then produced descriptive statistics on these 
behaviors for each role group (teacher, teacher leader, and administrator), school level 
(elementary, K-8, middle, 6-12, and high), cohort, and content area subgroup. We also 
compared the average engagement in these key activities across two groups: teachers who 
implemented modules as measured by the presence of uploaded student work and those 
teachers for whom we do not have evidence of module implementation. The results on these 
overall metrics of engagement are reported in Chapter 4. 


CoreTools data were also used to analyze to what extent (a) teachers were exposed to 
instructional cycles of LEARN content; (b) coaches provided feedback via commenting and peer 
review functions; (c) teachers edited key portions of modules; and (d) teachers uploaded 
student work (a proxy for classroom implementation). Results for these indicators are reported 
in Chapter 6. 


Modules 


During the first year of the evaluation we adapted the existing module rating rubrics 
(Reisman et al., 2013) that were developed for earlier studies of LDC. The rubrics, presented in 
Appendix D, were designed to examine the instructional quality and coherence of the LDC 
modules, and to address the rigor in both content and literacy development materials (i.e., 
template task, student work samples, and descriptions of the pacing and goals of the modules). 
The first two dimensions examined the quality of the teaching task, while the remaining four 
dimensions focused more holistically on module quality: (a) effective writing task; (b) alignment 
to the CCSS and local and state literacy and content standards; (c) fidelity to LDC module 
instruction; (d) quality instructional strategies; (e) coherence and clarity of module; and 
(f) overall impression. 


Each of the dimensions was rated using a 5-point scale with anchor points on the first five 
dimensions ranging from not present or realized to fully present or realized and the final 
dimension ranging from inadequate to advanced LDC module implementation. This scale 
mimics the structure of the three-level LDC jurying system scale that uses the ordered anchors 
of work in progress, good to go, and exemplary quality. For each dimension, a 1 represented 
the lowest possible level of quality, while a 5 represented the highest level of quality (see 
Appendix D). 


LDC Administrative Records 


The fidelity matrix analyses in Chapter 6 utilize administrative records collected by LDC as 
part of their ongoing program management. These records include (a) PLC reflection forms, and 
(b) attendance records for in-person meetings organized for the benefit of school 
administrators and teacher leaders. The PLC reflection form captures attendance at regular PLC 
meetings, the amount of time spent on LDC during these meetings, whether teacher leaders 
had a separate planning call with coaches, and the subjects that PLCs were working on during 
sessions. 


Administrative Data Used in Student Outcomes Analysis 


Student-level variables utilized in the outcome analysis included race/ethnicity, gender, 
poverty status, special education status, English language proficiency, gifted status, grade, and 
baseline achievement in mathematics and English language arts (ELA), as well as outcome year 
achievement in ELA on state assessments. Teacher-level indicators obtained and utilized 
included years of teaching experience and teacher attendance. We also requested and received 
roster files that establish a link between teachers and students via specific courses. 


2.2 Study Population and Generalizability 


Here we describe the population of schools and teachers who participated in LDC, how 
that population changed over time, and how that population compared to all schools and 
teachers in the district. Over the course of the project, two cohorts of schools were recruited to 
participate in LDC, with one cohort beginning implementation in 2016-2017 and a second 
cohort beginning in 2017-2018. The initial 20 Cohort 1 schools included 11 elementary schools, 
four middle schools, one high school, two K-8 schools, one 6-12 school, and one K-12 school. 
Cohort 2 included 20 elementary schools, eight middle schools, two K-8 schools, and one 6-12 
school. As displayed in Table 2.2, there was substantial attrition at both the school and student 
levels for each cohort. Nearly one third of Cohort 1 schools dropped out of the program after 
2016-2017, and within the remaining Cohort 1 schools, nearly half of teachers did not continue 
with LDC in 2017-2018. Attrition in Cohort 2 was somewhat less pronounced but still very 
substantial. About one quarter of schools did not continue with LDC in 2018-2019, and almost 
40% of the original Cohort 2 teachers left the program. 


Table 2.2 
Attrition of Cohort 1 and Cohort 2 Teachers, Administrators, and Schools 


Sample Teachers Administrators Schools 


Cohort 1 (beginning 2016-2017) 


Participated in LDC in 2016-2017 154 34 20 
Remained in LDC in 2017-2018 56 20 14 
Attrition rate as of 2017-2018 64% 41% 30% 
Remained in LDC in 2018-2019 29 14 11 
Attrition rate as of 2018-2019 81% 59% 45% 


Cohort 2 (beginning 2017-2018) 


Participated in LDC in 2017-2018 199 42 31 
Remained in LDC in 2018-2019 122 27 23 
Attrition rate as of 2018-2019 39% 36% 26% 


To better understand teacher and school attrition, we conducted an interview study of 
school administrators in the 2017-2018 school year. Twenty principals and assistant principals 
from both the West Coast district and our parallel New York City Department of Education 
study volunteered to be interviewed. The principal interview responses revealed a diversity of 
reasons. The main reason for teacher retention involved teacher and grade-level team 
decisions to leave or stay, followed by the principals’ decision to switch participation of 
teachers between the 2 years. Buy-in also seemed to affect teachers’ decisions on whether to 
continue. Although the sample was small, one of the reasons schools seemed to have dropped 
out was because they were overburdened by other priorities, including other reform efforts in 
the districts. Information submitted by school administrators to LDC confirmed that there were 
many other reform initiatives happening in schools during the study, including programs related 
to ELA, such as Early Language and Literacy Plans, Scholastic Leveled Bookrooms, and Achieve 
3000. See Appendix E for a detailed description of the results as well as the methodology for 
the interview study. 


In Table 2.3, we provide a snapshot of participation in 2018-2019, the final 
implementation year, by cohort, the year the teacher or administrator began participating in 
LDC, and school level. About two thirds of participants were from elementary schools, and 
about three quarters were from Cohort 2 schools. Both cohorts of schools also had teachers 
and administrators join LDC after the first year of implementation. Fifty-five teachers and 
administrators in Cohort 1 schools joined LDC in either 2017-2018 or 2018-2019 (18 of the 
participants who joined in 2017-2018 left after that year). Likewise, 83 teachers and 
administrators in Cohort 2 schools joined LDC in 2018-2019. These participants are not part of 
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our primary impact analyses. We planned to examine the impact of these late joining teachers 
in our supplementary impact analyses, but did not end up carrying them out because of 
insufficient teacher and/or student sample sizes. 


Table 2.3 
Number of Participants in 2018-2019 by Cohort/Start Year and School Level 


Cohort/start year Elementary K-8 Middle 6-12 High Total 
Cohort 1 school/started 2016-2017 27 0 13 0 3 43 
Cohort 1 school/started 2017-2018 9 0 5 0 2 16 
Cohort 1 school/started 2018-2019 12 0 5 0 4 21 
Cohort 2 school/started 2017-2018 92 16 26 15 0 149 
Cohort 2 school/started 2018-2019 63 5 9 6 0 83 
Total 203 21 58 21 9 312 


Next we describe how the participating schools, teachers, and students in the LDC 
program compared to those in the school district as a whole. In Table 2.4, we provide a 
snapshot for each of the 3 study years. For each year, we present characteristics of schools and 
their Grades 3-8 student populations compared to all schools and all Grades 3-8 students in 
the district. These calculations omit high schools and K—2 schools, which are not the focus of 
the LDC program, and for which we did not receive data from the district. 


Not surprisingly given LDC’s focus on middle schools, the LDC treatment group had a 
greater proportion of middle schools and a smaller proportion of elementary schools than the 
district as a whole. In LDC schools, there were more students who were Black and Hispanic and 
fewer students who were White and Asian/Pacific Islander than the district’s overall student 
population. Students in LDC schools were slightly more likely to be in poverty, and slightly less 
likely to be in gifted programs than students districtwide. Students were classified as limited 
English proficient and eligible for special education services in similar proportions in the two 
groups. LDC schools were slightly lower performing in both ELA and math than all schools in the 
district serving students in Grades 3-8. 
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Table 2.4 


Characteristics of LDC Schools and Grades 3-8 Student Population at LDC Schools Compared to LAUSD as 
a Whole 


2016-2017 2017-2018 2018-2019 
Whole Whole Whole 
Variable LDC district LDC district LDC district 
School level 
Elementary schools (%) 55 79 65 78 68 78 
Elementary/middle schools: K-8 (%) 10 2 5 3 6 3 
Middle schools (%) 25 14 28 14 24 14 
Middle/high schools: 6-12 (%) 5 3 2 4 3 4 
K-12 schools (%) 5 1 0 2 0 2 
Student-level characteristics 
Hispanic (%) 82 73 80 73 77 72 
Black (%) 12 8 9 8 10 8 
Asian/Pacific Islander (%) 2 7 4 6 4 6 
White (%) 4 11 6 11 7 11 
Female (%) 50 49 49 49 49 49 
Limited English proficient (%) 20 19 16 16 17 18 
Special education (%) 12 13 12 14 11 12 
Gifted (%) 13 15 13 15 14 15 
In poverty* (%) 64 62 89 82 87 84 
Student achievement 
Mean standardized ELA achievement -0.12 0.00 -0.09 0.00 -0.08 0.00 
Mean standardized math achievement = -0.14 0.00 -0.09 0.00 -0.06 0.00 


Next we explore how the main LDC teacher groups of interest compared to teachers 
districtwide (see Table 2.5). These groups are Cohort 1 teachers with 2 years of LDC 
participation as of 2017-2018 and Cohort 2 teachers with 2 years of LDC participation as of 
2018-2019. We report on a number of teacher characteristics, as well as the mean 
achievement of these teachers’ students in the baseline year. Not all variables were available 
for each of the two cohorts/years. For example, demographic characteristics were not available 
for 2017-2018. The analysis suggests that Cohort 1 teachers were a little less likely than 
teachers in the district as a whole to have 3 or fewer years of experience. Cohort 1 teachers 
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also taught students that were lower performing on average at baseline than Grades 3-8 
students in the district as a whole. On average there were more Cohort 2 teachers who were 
Hispanic or Black and fewer Cohort 2 teachers who were White than in the district as a whole. 
Cohort 2 teachers taught students that were somewhat higher performing on average at 
baseline than students in the district as a whole. 


Table 2.5 


Characteristics of LDC Core Content Area Teachers With 2 Consecutive Years of LDC Participation and 
Districtwide Teachers, by Cohort/Year 


Cohort 1 in Cohort 2 in 
2017-2018 2018-2019 
Whole Whole 
Variable LDC district LDC district 
Hispanic teachers (%) N/A N/A 48 41 
Black (not Hispanic) teachers (%) N/A N/A 17 12 
Asian (not Hispanic) teachers (%) N/A N/A 10 12 
White (not Hispanic) teachers (%) N/A N/A 20 35 
Female teachers (%) N/A N/A 67 68 
Teachers with 3 years of experience or less (%) 8 12 16 15 
Teachers with regular/permanent assignment (%) N/A N/A 86 82 
Teacher attendance rate 97 96 93 94 
Mean standardized ELA achievement at baseline of -0.40 -0.18 -0.02 -0.18 
students taught by teachers 
Mean standardized math achievement at baseline -0.40 -0.18 0.03 -0.18 


of students taught by teachers 


2.3 Sample for Implementation and Outcomes Analyses 


Here we explore the teacher and administrator samples for the various implementation 
analyses in the final study year (2018-2019), as well as for outcomes analyses across the 
multiyear study. Table 2.6 displays the overall population and sample sizes for the different 
study measures and analyses included in this report. This includes all participants in 2018-2019. 
As can be seen, data were available for a large majority of the participants. Ninety-three 
percent of teachers consented to participate in the study, with 88% of all teachers completing 
the survey in spring 2019. All administrators consented to participate in the study, and 85% 
completed the survey. All teachers (100%) and nearly all administrators (96%) were present in 
the CoreTools dataset, which was provided directly to CRESST by LDC, and did not depend on 
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teachers’ individual study consents. Nearly all teachers (94%) were present in the PLC reflection 
attendance data as well (three schools did not properly record the data and therefore there are 
no attendance data on teachers at these schools). 


Table 2.6 


Number of Participating Schools and Teachers in the District and Sample Size for Different Instruments, 
2018-2019 


Instruments Classroom teachers Administrators Schools 
Participated in LDC 264 48 34 
Consented to CRESST study 246 48 34 
Completed survey 231 41 33 
Present in CoreTools dataset 264 46 34 
Authored/coauthored a module with 194 N/A 33 
student work uploaded 
Present in PLC reflection form 247 35 31 
attendance records 
Leader in-person meeting attendance N/A N/A 34 
Included in outcomes analysis for 104 N/A 25 
2018-2019 


The school district required individually signed consent forms before releasing teacher 
data and teacher/student rosters, so for the outcome analysis, we only received data on 
teachers who consented to participate in the study. Samples were further restricted by the 
need for student achievement data for both the outcome year (2017-2018 or 2018-2019 for 
Cohorts 1 and 2 respectively) and the baseline year (2015-2016 or 2016-2017 for Cohorts 1 
and 2 respectively). As a result, participants teaching either in high school or the primary 
elementary grades (K-3) were not included in any of the student outcome analyses. Middle 
school teachers who did not teach a core ELA, science, or social studies/history class were also 
excluded from the analyses. Of the 264 teachers from 34 schools who participated in LDC in 
2018-2019, 104 teachers from 25 schools were represented in at least one of the quasi- 
experimental analyses reported on in Chapter 7. 


In Tables 2.7 and 2.8, we list all the potential teacher samples by cohort, school level, start 
year, and outcome year. Our previous reports examined the impact of LDC after 1 year for 
Cohort 1 and Cohort 2 elementary and middle school teachers. We also previously estimated 
the impact of Cohort 1 middle school teachers after 2 years of implementation. This final 
annual report includes models pooling across cohorts for each school level separately, and for 
the elementary and middle schools combined. We also examine the impact of LDC as 
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implemented in Cohort 1 schools by teachers with 3 years of LDC participation. We had also 
intended to conduct supplementary analyses examining the impact of teachers joining existing 
LDC PLCs in later years, but we did not have a sufficient number of teachers and students to 
conduct those analyses. 
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Table 2.7 


Number of Cohort 1 Classroom Teachers, Implementation Years, Schoo! Level, and QED Eligible Students 


Consented 
School level/ Outcome Years in Consented teachers with Summary of 
start year year LDC Teachers teachers eligible students reporting 
Elementary school 
2016-2017 2016-2017 1 87 80 32 2017 Report 
2016-2017 2017-2018 2 26 26 3 2019 Report® 
2016-2017 2018-2019 3 17 17 0 Not conducted 
2017-2018 2017-2018 il 22 21 1 Not conducted 
2017-2018 2018-2019 2 10 10 0 Not conducted 
2018-2019 2018-2019 1 ital 10 0 Not conducted 
Middle school 
2016-2017 2016-2017 1 41 40 35 2017 Report 
2016-2017 2017-2018 2 23 23 22 2018 and 2019 
Reports” 
2016-2017 2018-2019 3 9 9 8 2019 Report 
2017-2018 2017-2018 1 6 6 4 Not conducted 
2017-2018 2018-2019 2 5 5 5 Not conducted 
2018-2019 2018-2019 1 5 4 4 Not conducted 
High school 
2016-2017 2016-2017 il 26 25 0 Not applicable 
2016-2017 2017-2018 2 4 4 0 Not applicable 
2016-2017 2018-2019 3 3 3 0 Not applicable 
2017-2018 2017-2018 1 7 6 0 Not applicable 
2017-2018 2018-2019 2 1 il 0 Not applicable 
2018-2019 2018-2019 1 3 2 0 Not applicable 


“There was insufficient sample size to test the impact of Cohort 1 elementary teachers in their second 
year of implementation independently, but these teachers were included in pooled analyses examining 
the impact of Cohort 1 and 2 elementary teachers after 2 years of implementation. 
>This sample was reported on independently in the 2018 report, and also as a part of the pooled 
analyses in this report. 
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Table 2.8 


Number of Cohort 2 Classroom Teachers, Implementation Years, Schoo! Level, and QED Eligible Students 


Consented 
School level/ Outcome Years in Consented _ teachers with Summary of 
start year year LDC Teachers teachers eligible students reporting 

Elementary school 

2017-2018 2017-2018 il 137 129 85 2018 Report 

2017-2018 2018-2019 2 78 77 31 2019 Report 

2018-2019 2018-2019 1 63 57 16 Not conducted 
Middle school 

2017-2018 2017-2018 1 56 53 43 2018 Report 

2017-2018 2018-2019 2 38 36 32 2019 Report 

2018-2019 2018-2019 1 12 9 8 Not conducted 
High school 

2017-2018 2017-2018 il 6 5 0 Not applicable 

2017-2018 2018-2019 2 6 5 0 Not applicable 

2018-2019 2018-2019 1 3 3 0 Not applicable 


In addition to the CoreTools analytic files, we also received module artifacts from LDC 
teachers. We restricted our analysis to modules that were created during the 2018-2019 school 
year and included original uploaded student work examples, because these student examples 
were considered a proxy for implementation in the classroom and were required for module 
scoring. That restriction yielded a pool of 209 modules that were authored or coauthored by 
194 teachers (nearly three quarters of all participating teachers) across 33 schools. To align with 
our research questions and to have analyzed modules be representative of the teacher sample, 
we sampled the last module created by each teacher wherein they served as the sole or lead 
author. Our intent was to represent teachers’ best LDC work, and presumably, modules created 
later in the school year would be more skilled than those created earlier. The total sample was 
155 modules. Table 2.9 provides further detail about the distribution of modules. We also 
conducted an exploratory analysis of the 40 elementary teachers and 27 secondary teachers 
who created a complete module both in 2018-2019 and in 2017-2018. 
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Table 2.9 
Background Variables for the Primary Module Analysis 


Elementary Secondary Total 
Variables # % # % # % 

Cohort/start year 

Cohort 1 (2016-2017) 16 15.4 4 7.8 20 12.9 
Cohort 1 (2017-2018) 3 2.9 6 11.8 9 5.8 
Cohort 1 (2018-2019) 4 3.8 2 3.9 6 3.9 
Cohort 2 (2017-2018) 51 49.0 26 51.0 77 49.7 
Cohort 2 (2018-2019) 30 28.8 13 25.5 43 27.7 
Subject 

ELA 38 36.5 32 62.7 70 45.2 
Science 43 41.3 3 5.9 46 29.7 
Social studies 23 22.1 16 31.4 39 25.2 
Total 104 67.1 51 32.9 155 100.0 


2.4 Survey Recruitment and Administration 


As in previous years, CRESST and LDC worked together to conduct the consent process for 
teachers and administrators who newly joined LDC in 2018-2019, and were successful in 
consenting 88% of these new participants. As required by the UCLA Office of the Human 
Research Protection Program, all consent forms included language stating that the study was 
voluntary, that data would be protected, and that by signing the form, participants consented 
to be emailed an electronic survey in spring 2019 and to have CRESST request district data 
linking them and their students. All participating teachers, teacher leaders, and administrators 
were also compensated with a $50 gift card after completing the survey. 


Surveys were administered in spring 2019. CRESST coordinated with LDC staff and coaches 
to administer the online surveys during common planning sessions. Multiple email reminders 
were sent to participants who were not present at these sessions or who otherwise did not 
complete the survey. The teacher survey administration was closed at the end of the school 
year in June 2019. The administrator survey window was left open until early August when they 
returned from their summer break. 


2.5 Module Rating Process 


LDC requirements specified that all teachers implement a minimum of two modules over 
the course of the year, with the first spanning one week based on one text and the second 
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spanning across multiple weeks and incorporating multiple texts. Modules could be developed 
as original units of work or could be adapted from existing modules within the LDC CoreTools 
Library. Modules could also be either developed or adapted in solo or collaboratively with other 
teachers within the PLC. LDC modules were collected from elementary and secondary teachers 
who participated in LDC during the 2018-2019 school year. In total, 155 modules were rated 
with 50 or 32.3% being rated by two panelists. Further details about the modules can be found 
in Appendix F. 


Eight expert raters with experience teaching in the targeted grade spans and content 
areas were recruited from Los Angeles County schools. Four panels were convened with two or 
three experts rating each of the following sets of modules: elementary ELA and social science, 
elementary and secondary science, secondary ELA, and secondary social science. It should be 
noted that one rater served on both the secondary ELA and the secondary social science panels. 
In addition, three of the teachers had 2 years of prior experience working as raters on the LDC 
module analyses and three had 1 year of prior experience. 


Separate trainings lasting approximately 2 hours were conducted for each module scoring 
panel. All trainings were conducted by a member of the evaluation team who is an expert on 
the Common Core and the rating of student and teacher artifacts, and who had been 
conducting the training for the past three cycles of scoring. The training included an overview of 
the LDC goals, template task, the structure of the modules, and the CRESST rating dimensions. 
Once the training was complete, calibration was conducted by having teachers individually 
score and then discuss their ratings for one module in the content area on which they would be 
focusing. Scoring was then conducted on subsequent days with each module individually rated. 
Those modules rated by two expert teachers were then discussed with the goal, but not the 
requirement, to reach consensus (see Carlson & McCaslin, 2003). All discussions were 
facilitated by the same evaluation team who conducted the initial training. 


2.6 Analytical Approaches 


Both quantitative and qualitative analytic methodologies were applied to the data to 
answer the evaluation questions about how LDC was implemented, conditions affecting 
implementation, and program impacts. The following describes the approaches used to analyze 
each dataset. 


Surveys 


Survey responses were analyzed using descriptive statistics for multiple-choice items and 
qualitative coding for open-ended responses. As previously noted, surveys were administered 
to teachers, teacher leaders, and administrators. Some teacher leaders were classroom 
teachers while others were coaches or coordinators. Because of this, the samples were not 
mutually exclusive, with teacher leaders who taught in the classroom represented in both the 
teacher and teacher leader results. Generally, we report the number and percentage of 
respondents who selected different multiple-choice options. In a few cases, where responses 
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are numerical rather than categorical, we present means rather than proportions. Descriptive 
statistics for all multiple-choice items are presented in full in Appendix A for teachers, Appendix 
B for teacher leaders, and Appendix C for administrators. 


LDC CoreTools 


The first stage of our analysis examined the proportion of all participants who created 
CoreTools user accounts, and engaged in three key behaviors: viewing, editing, and 
commenting on modules. We then analyzed each of the three key measures of participants’ 
interaction with the LDC online tools, and reported frequencies and/or descriptive statistics 
(e.g., minimum, maximum, mean, standard deviation) as measures of participants’ engagement 
with the online LDC system. In addition to reporting the overall results, we provide results by 
cohort, the content areas taught (ELA, social studies/history, and science), by participant role 
(teacher, teacher leader, and administrator), and school level (elementary, K-8, middle, 6-12, 
and high) whenever feasible. Finally, we explored the difference in CoreTools engagement 
between two groups of teachers: those who completed and taught modules and those who did 
not appear to complete the design and implementation process. 


Modules 


We used descriptive statistics (means, standard deviations, and percentages) to analyze 
overall and subgroup performance for each content area, cohort, and school level on each of 
the six dimensions. This included analyses of the modules submitted during the 2018-2019 
school year, as well as an exploratory analysis examining results across time for teachers who 
submitted modules during the last 2 school years. Additionally, generalizability theory (G 
theory) was used to examine potential sources of error during the rating process to help 
determine the validity of the scores as well as the construct validity of the rubrics (see 
Shavelson & Webb, 1991). Finally, teacher comments during the debriefings were examined to 
determine other potential issues with the rubrics and/or rating process. 


Fidelity of Implementation Analysis 


To examine fidelity of implementation we conducted descriptive analyses of a variety of 
data sources, including a teacher survey, CoreTools data, PLC reflection data, and attendance 
records for administrator and teacher leader in-person meetings. As outlined in the fidelity 
matrix (see Appendix G), fidelity levels were set at the teacher, module, school, and program 
levels. The process for most indicators involved several levels of aggregation, with, for example, 
module editing being measured at the teacher level, school level, and program level. School- 
and program-level scores were typically based on the proportion of teachers or modules that 
met the threshold for adequate implementation. 


An analysis of the relationship between fidelity of implementation and outcomes was also 
conducted. This analysis utilized the analytic sample from our primary quasi-experimental 
investigation of the impact of LDC as practiced by teachers with 2 years of LDC experience. Two 


20 


models produced school and teacher residuals respectively. These residuals were then used to 
divide the sample into high-, medium-, and low-achieving schools and teachers. Descriptive 
analysis and one-way ANOVA were used to examine whether mean fidelity scores differed for 
the three different achievement groups. Additional information on methodology can be found 
in Chapter 6, Section 7. 


Student Outcomes: Matching and Intervention Exposure 


Here we describe (a) our approach to selecting matched schools and students for our 
quasi-experimental design analyses estimating the effect of LDC on student Smarter Balanced 
Assessment Consortium (Smarter Balanced) ELA assessment scores, and (b) our approach for 
modeling students’ exposure to LDC and non-LDC instruction in core content classes. 


For all our analyses, we used a two-step matching process to identify a reduced pool of 
comparison students and teachers at schools with similar characteristics to the schools in the 
intervention sample. To accomplish this, we first identified up to five of the most similar 
comparison schools for each intervention school based on a Euclidian distance measure, by 
using the nearest neighbor analysis option in SPSS 24.0 (see Fix & Hodges, 1951; Wang et al., 
2007). The variables used in this process were the percentage of students eligible for free or 
reduced-price lunch, the percentage of Black students, mean baseline student achievement in 
ELA, mean baseline student achievement in mathematics, the average attendance rate of 
teachers, the percentage of teachers with 3 or fewer years of teaching experience, and the 
school grade span where feasible. We generally used all five identified comparison schools to 
establish the potential matching pool, but for selected LDC schools we only used four of the 
initial five matches. Once the pool of comparison schools was identified, their students and 
teachers were also identified, and student-level matching was conducted so that the resulting 
sample would resemble the type of sample one would expect to obtain through random 
assignment. 


The student-level matching technique we employed was coarsened exact matching (CEM) 
(lacus et al., 2011). CEM is a flexible matching approach with many favorable properties and 
allows the researcher to specify the precise conditions under which students are matched. For 
categorical variables, such as race/ethnicity or free or reduced-price lunch status, this can entail 
exact matching, while for continuous measures, such as baseline individual student 
achievement and aggregate class-level achievement, cut-points for matching can be specified. 
With this approach we were able to set precise cut-points on the most important baseline 
indicators, such as baseline academic achievement, to ensure that where possible every 
treatment student was matched with a suitable comparison. Student matching variables we 
used in CEM included Hispanic, Black, poverty status, female, English language proficiency 
(English language learner), special education status, gifted status, mean baseline achievement 
in mathematics and ELA, and grade level. 
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During matching we also included a few variables capturing information on the teachers 
and peers to whom students were exposed. These variables included mean baseline ELA 
achievement of the student’s peers in his/her core content classes, and the percentage of core 
subject teachers with 3 years of teaching experience. 


The typical structure of course taking at the middle school level involves students 
potentially being exposed to multiple teachers, with each responsible for a different core 
content class. Specifically, middle school students in the study had exposure opportunities 
across three content areas (ELA, social studies/history, and science) taught by intervention 
teachers. As a result, students were not necessarily nested under individual teachers, but 
instead were likely to have connections to multiple teachers in the available time period prior 
to each testing outcome (students at the elementary school level were also sometimes exposed 
to multiple teachers but to a lesser extent). Therefore, LDC effects were estimated using an 
extension of the standard multilevel modeling framework known as multiple membership 
multiple classification (MMMC) models (Browne et al., 2001). 


These models can account for complex classification structures, such as the LDC context, 
in which students are nested within schools but are also members of multiple classes led by 
different teachers who may or may not be implementing LDC. MMMC has the flexibility to 
account for this type of complex nesting structure in which students are hierarchically nested 
under schools but may have one-to-many relationships with teachers. There are three 
classification levels in the models we employ: students, teachers, and schools. In the MMMC 
modeling approach, each observation at the lowest level represents one student. Figure 2.1 
shows how students can be exposed to teachers in different content areas, who may or may 
not be participating in LDC. Our MMMC modeling approach can account for this complicated 
structure. 
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Figure 2.1 


Variation in Student Exposure to Teachers 


Teacher 1 (ELA) 


Teacher 2 (History) Teacher 3 (Science) 


LESTNS< | 


Student 1 Student 2 Student 3 Student 4 Student 5 Student 6 


In the MMMC modeling approach, each observation at the lowest level represents one 
student. The weight each teacher receives with respect to each student is dependent on the 
student’s exposure to his or her teachers in each of the three core content areas. The total 
student exposure adds to a unity (i.e., a possible total exposure of 1) across their courses in the 
three content areas in a given school year. While this general weighting approach applies to 
both elementary and middle schools, the course structure of the datasets required us to use 
different weighting procedures for elementary and middle schools. 


In both the elementary and middle school datasets, students were linked to teachers 
through statewide course identifiers and accompanying terms in which each student was 
enrolled in a given course under a specified teacher. For elementary students, the terms 
consisted of three marking periods. For middle school students, the terms consisted of two 
semesters. Elementary students were required to have received marks in each of the three 
marking periods, and middle school students were required to have received grades in ELA 
courses in both semesters. Students could potentially have data records connected to multiple 
teachers covering varying time periods in the same course. For each of the three core content 
courses we then collapsed the links into a single measure of number of terms of potential 
content exposure. 


In elementary school, in the event that a student was exposed to more than one teacher, 
each marking period was given equal weight in distributing teacher/student exposure. For 
example, if a student was enrolled in two marking periods under one teacher, then that teacher 
was coded as .67 for having contributed to two thirds of the students’ core curriculum 
exposure. If the same student enrolled in the third marking period with a different teacher, 
then that third marking period teacher would have been coded as .33 and all other teachers in 
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the sample would have been coded as zero. This would then result in the student’s exposure 
adding to a unity (1). 


In middle school, students’ exposure to teachers at the course level in the three core 
content areas was coded in the same manner as in the elementary grades, but based on the 
two semesters. A difference in our middle school coding process was that we did not force each 
core content area into equal weighting. Instead each core content area exposure contributed to 
a core content area total sum that formed the basis from which the weights were proportioned. 
Most commonly, a student had equivalent core instruction exposure in each of the three 
content areas (often two terms each). In that scenario, if a student had exposure to three 
different teachers, then each teacher would contribute one third (.33) of the overall core 
curriculum exposure and all other teachers in the sample would be coded as zero. However, in 
seventh grade the students often only had one semester of science. It was also occasionally the 
case that students would take both a core and an additional science course; in these cases, 
both science courses were included in the LDC analysis. 


The weighting in middle school was always distributed as a proportion of the total 
semesters across the three content areas. Therefore, if a student accumulated one science unit 
(one semester), two social studies units (two semesters), and two ELA units (two semesters), 
the base number of units would be five. Using that scenario, the science teacher would 
contribute one fifth (.20) of the overall core curriculum exposure with the social studies and 
science teachers contributing two fifths (.40) each, again resulting in the student’s exposure 
adding to a unity. Tabular versions of the above examples can be found in Appendix H. 


For this study, we modeled the treatment intervention variable as a fixed effect at the 
student level in two different ways. The first dosage-dependent approach takes into account 
the students’ level of exposure to the intervention teachers. In this approach, the treatment 
was structured as a continuous response variable, coded as zero for comparison students and 
coded as a positive value for treated students, albeit never exceeding one. The positive value 
assigned to treated students in the dosage-dependent approach was simply the sum of the 
intervention teacher weights linked to the treated student. The second approach was dosage 
independent and classified any student exposed to an intervention teacher via at least one 
course as a treated individual. In this approach the treatment variable was dichotomous: coded 
as one for treated students and zero for comparison students. 


Student Outcomes: Analysis Approaches 


As with other multilevel models, MMMC accounts for the nonindependence of 
observations within cluster by adjusting the inferences on parameter estimates for the 
correlations between responses in a cluster. This modeling approach, however, becomes 
computationally cumbersome using traditional frequentist estimation methods. As 
recommended by Browne et al. (2001), to address this issue we employed Bayesian methods 
using Markov chain Monte Carlo (MCMC) techniques. Multilevel models incorporate 
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demographic and achievement variables used in the matching design as covariates, making the 
findings “double robust” in that characteristics can be controlled for in both matching and 
outcomes analysis stages. Student demographic and baseline achievement variables that were 
used in the matching process were also included as covariates in the MMMC model. The full 
specifications for both models can be found in Appendix H. 


In Chapter 7, we report on both the primary and supplementary analyses of the impact of 
LDC. Primary analyses focus on impacts after 2 years of teacher participation in LDC (for both 
Cohorts 1 and 2), and impacts after 3 years of teacher participation in LDC (for Cohort 1 only). 
Data from both cohorts were pooled for the primary analyses measuring impact after 2 years of 
implementation to increase statistical power and to produce more precise impact estimates. 
For the pooled analyses, the outcome year for Cohort 1 was spring 2018 and for Cohort 2 was 
spring 2019. To maximize the sample, we also conducted a primary analysis pooling schools, 
teachers, and students across both cohorts (Cohort 1 and Cohort 2) and school levels 
(elementary and middle). For this, we made the assumption that being exposed to LDC in all 
core content areas at the middle school level was equivalent to being exposed to a single 
teacher implementing LDC at the elementary level. This analysis was specified using only the 
dosage-independent version of the treatment variable. 


The other primary analysis reported on in Chapter 7 examines the impact of LDC as 
implemented by Cohort 1 teachers with 3 years of LDC experience on student ELA scores in 
spring 2019. This analysis focuses exclusively on middle school students due to the lack of 
achievement scores for elementary school students in the 2015-2016 baseline year. We also 
report supplementary analyses examining (a) the impact of Cohort 2 teachers with 2 years of 
LDC experience (these groups of elementary and middle school teachers are subsamples of the 
pooled group of teachers in our primary analyses); and (b) the impact of LDC on subgroups of 
students broken down by whether they were exposed to LDC in the prior year, and how much 
LDC instruction they were exposed to in the outcome year. Analyses examining groups of 
teachers that joined existing LDC PLCs in later years were not conducted due to insufficient and 
limited teacher and/or student samples. 
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3.0 Survey Analysis 


In this chapter we present the survey results. First, we summarize teachers’ responses. 
We then summarize the teacher leaders’ responses, followed by the administrators’ responses. 
Within each of these sections, we organize results by the following domains: LDC participation; 
professional learning community and teacher collaboration; LDC training and support; LDC 
module implementation; leadership support; impact; and issues of scale-up and sustainability 
facilitators and barriers. The next section summarizes the open-ended responses regarding 
supports for implementation of LDC. We then present results of an exploratory analysis 
examining results across time (2018 and 2019) for three scales in which teachers rate the 
impact of LDC on themselves and their students. We end with a summary of results. 


We use abbreviations to identify which participants answered specific questions for each 
domain. We preface teacher items with “T,” teacher leader with “TL,” and administrator 
questions with “A.” For example, LDC Participation (T1—4) indicates that teacher survey items 
1-4 are used to provide information on LDC participation. Survey questions and descriptive 
results are presented in full in Appendix A for teachers, Appendix B for teacher leaders, and 
Appendix C for administrators. 


3.1 Teacher Survey Results 


As noted earlier, 231 teachers spanning 33 schools completed the survey. Among the 
teachers who completed a survey, the largest proportion taught an elementary school grade 
while the smallest proportion taught at the high school level. More specifically, 65% of teachers 
taught in 22 elementary schools, 7% in two K—8 schools, 18% in 7 middle schools, 8% in one 
Grades 6-12 school, and 2% in one high school (see Table 3.1). When examining participation 
by cohort, 47 (20%) of the teachers were from Cohort 1 and 184 (80%) were from Cohort 2. 
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Table 3.1 
Number of Schools and Teachers Completing the Survey in 2017-2018 


Cohort and start year Elementary K-8 Middle 6-12 High Total 


Number of schools 22 2 7 1 1 33 


Number of teachers 


In Cohort 1 schools and started 2016-2017 13 0 8 0 2 23 
In Cohort 1 schools and started 2017-2018 4 0 4 0 2 10 
In Cohort 1 schools and started 2018-2019 10 0 4 0 0 14 
In Cohort 2 schools and started 2017-2018 67 12 20 14 0 113 
In Cohort 2 schools and started 2018-2019 55 5 6 5 0 71 
Total 149 17 42 19 4 231 


Among the 231 teachers who completed the survey, 149 reported teaching in an 
elementary school, and 82 reported teaching in a secondary school. The secondary teachers 
reported teaching one to seven classes (M = 4.78). In addition, the secondary teachers reported 
that they used LDC materials in zero to six classes (M = 2.86).” As shown in Figure 3.1, the 
distribution of LDC across content areas was 42% in ELA, 31% in history/social studies, 18% in 
science, 6% in other areas (e.g., health, Spanish), and 3% in mathematics. 


One teacher reported not using LDC materials in any classes. 
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Figure 3.1 


Secondary Teachers’ Content Areas Implemented 


Content Areas Implemented By Secondary Teachers 
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LDC Participation (T1-T4) 


Approximately one third of responding teachers (34.6%) reported that 2018-2019 was 
their first year of implementing LDC, with the remaining two thirds (65.4%) having 1 or 2 years 
of previous experience. Among those who were returning to LDC, a great deal of variation was 
reported about their module implementation prior to the 2018-2019 school year (i.e., 
2016-2017 and/or 2017-2018). More specifically, teachers reported implementing about two 
modules (M = 2.19, range = 0 to 10) and three mini-tasks outside of the context of a module 
(M = 2.78, range = 0 to 20) prior to 2018-2019. 


Six of the seven teachers who did not participate in a PLC did report using LDC tools in 
their instructional planning or classroom instruction during the 2018-2019 school year. The 
most common activities for these six teachers involved using CoreTools to access existing 
modules or mini-tasks (n = 5) and design modules or mini-tasks (n = 5). Others stated that they 
received modules or mini-tasks from their coworkers (n = 4) or that they took an LDC online 
course (n = 3). Finally, one of the teachers reported that they did not implement because “My 
modules didn’t have tools that | needed.” 


Professional Learning Community and Teacher Collaboration (T5—T9) 


The vast majority of teachers (96%) participated in a PLC that at least partly focused on 
implementing LDC. Most of these teachers reported that their PLC met every other week (53%) 
or once a month (31%). The most common reasons stated for not meeting every week included 
other priorities that competed with LDC (43%), PLC time not being protected (17%), and limited 
interest in meeting (11%). The frequency of conversations about LDC that reportedly took place 
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outside of the PLC meetings typically ranged from once a week to once a month (84%). About 
two thirds of respondents (66%) stated that PLC meetings typically lasted 45-59 minutes with 
another quarter (26%) stating that they lasted 60-74 minutes. 


LDC Training and Support (T10-T13) 


Approximately 90% of the teachers who responded stated that their PLC was moderately 
to very effective in all aspects listed on the survey. This included creating a comfortable working 
environment, fostering an environment for sharing instructional plans, helping teachers 
improve their LDC instructional plans, and allowing space for the sharing of student work. 


Teachers were asked their opinions of the online course materials in the LEARN tab in LDC 
CoreTools (see Figure 3.2). Overall, more than two thirds of the teachers rated the materials as 
good to excellent on each aspect. Overall, teachers were particularly enthusiastic about the 
clarity and relevance of the information provided, with more than four out of five teachers 
rating these aspects as good or excellent. On the lower end, about one third of teachers gave a 
rating of poor or fair concerning the ease of use of the online course materials in general and 
the videos specifically. 


Figure 3.2 
Teacher Perceptions of Online Course Materials That LDC Coaches Used or Directed Their Teachers to Use 
Online Course Materials (Teacher Survey) 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
Clarity of information presented. 
Relevance of information presented. 
Ease of use. 
Usefulness of resource documents. 


Usefulness of videos. 


Degree to which course materials helped teachers to 
create and/or adapt LDC modules. 


Opportunity to extend learning when needed or desired. 


@Poortofair Good to excellent 


Note. n= 224. 


Teachers were also asked to report on the supports that they received from their LDC 
coaches. When asked in general, almost all of the teachers reported that they were able to get 


feedback and support from their LDC coaches (96%) either through CoreTools or in the PLC, 
including the receipt of written feedback in a timely manner (94%) through CoreTools. Despite 
this, when asked to rate the usefulness of the coaching, 14% of teachers indicated that they did 
not receive written feedback from their coach within LDC CoreTools. Of those who did report 
receiving the different supports, views were positive with more than three fourths of teachers 
reporting them as moderately to very helpful (see Figure 3.3). These included written feedback, 
Zoom video conferences, email or phone communication, or other sources of feedback. 


Figure 3.3 
Teacher Perceptions of the Different Types of LDC Coaching Supports 


Helpfulness of LDC Coach Supports (Teacher Survey) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
Written feedback in LDC CoreTools in the comments areas MoyH 80% 
and/or via th teacher work rubric (n = 193) 
One-on-one Zoom video conference and/or call (n = 179) EE ltt————( 82% 
Email or phone communication (n= 166) (2277 ZSoo 
Other (n= 62) (27 ST 


@Notora little helpful @ Moderately or very helpful 


LDC Implementation (T14-T29) 


This domain covers questions on module creation (T14-17), module peer review 
(T27-29), and classroom implementation (T18-26). 


Module Creation (T14-T17) and Module Peer Review (T27-T29). Teachers on average 
adapted (M = 1.83, range = 0 - 11) twice as many modules as they created (M = 91, 
range = 0 - 15). New modules were most likely to be created by teams of two or more teachers 
(62%), although some respondents reported doing the work with their PLC as a whole (21%) or 
on their own (32%). Most teachers (79% to 93%) also reported that they were able to 
accomplish most of the module creation tasks to a moderate or great extent. As can be seen in 
Figure 3.4, teachers were most likely to feel successful (i.e., a moderate extent to a great 
extent) with regard to selecting standards and creating standards-driven writing assignments, 
and identifying the skills students need to develop to complete the writing assignment. The 
tasks that teachers reported doing the least (i.e., not at all or a little bit) included differentiating 
instruction and planning a variety of methods to assess student progress. 
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Overall, the formal review process for modules seemed to be underutilized. First, only one 
third of the teachers (34%) who responded to the survey reported attending a peer review or 
curriculum alignment workshop during the 2018-2019 school year. Second, over 60% of 
teachers did not submit any modules for the LDC National Peer Review process. Despite this, 
the 83 teachers who did submit for peer review tended to find the process moderately to very 
helpful (69%), and only 8% found the process to be not helpful. 


Figure 3.4 


Extent to Which Teachers Completed Various Module Creation Tasks 


Module Creation Tasks (Teacher Survey) 
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Classroom Implementation (T18-T26). Teachers seemed to implement fewer modules 
than they created or adapted. More specifically, teachers reported implementing about two 
modules (M = 2.16, range = O - 6) during the school year, and three additional mini-tasks 
outside of the context of a module (M = 3.33, range = 0 - 20). For many of the teachers 
(n = 144) this included implementing a “Find and Teach” module from CoreTools. More 
specifically, of those who did implement this type of module, the majority reported adapting 
the module (72%), with the remaining respondents implementing the module as it was found in 
CoreTools (28%). 
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Teachers were also asked to talk about later modules they created or adapted after 
completing the Find and Teach. In this case, about one third of teachers (31%) reported starting 
from a template in CoreTools with the remaining two thirds (69%) adapting an existing module 
found in the LDC Library. Furthermore, 87% of teachers reported that they had already 
implemented the module by the time they completed their survey. The remaining teachers 
were planning to implement later during the school year (7%), during the next school year (4%), 
or were not planning to implement at all (2%). 


Finally, teachers were asked to report on what they were able to accomplish when 
teaching their LDC modules (see Figure 3.5). Overall, the vast majority of teachers (84% to 92%) 
reported accomplishing each of the activities listed to a moderate or great extent. Respondents 
were most likely to report confidence in engaging students in understanding the assignment 
and its rubric (92%) and accessing complex text (91%). The area where teachers seemed to 
have the most area to grow involved the providing of feedback to students using rubrics, with 
2% reporting that they did not accomplish this at all and 14% reporting that they only 
accomplished this a little bit. 


Figure 3.5 
Activities Completed When Teaching LDC Modules 


Classroom Implementation (Teacher Survey) 
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Leadership Support (T33-T37) 


This domain covers questions on teacher leader support (T33), administrator support 
(T34-36), and teacher leadership role in LDC (T37). 
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Teacher Leader Support (T33). Teachers were asked to report on the support received 
from the teacher leaders in their respective PLCs. In this case, teacher opinions were 
overwhelmingly positive, with more than 90% agreeing with each of the five types of support 
listed, and more than half strongly agreeing. More specifically, about two thirds of the teachers 
indicated that the teacher leader effectively supported the PLC meetings and created an 
atmosphere where they were comfortable asking for feedback. In addition, more than half 
strongly agreed that the teacher leader helped them to align LDC to the broader school 
instructional goals, provided useful feedback on the modules, and effectively invited teachers 
to join LDC. 


School Administrator Support (T34—T36). According to teachers, administrators at many 
of the schools were engaged with the PLC. More specifically, 46% reported that their 
administrator attended the PLC meetings about three quarters or more of the time, 22% 
reported that their administrator attended about half of the meetings, and 32% reported that 
their administrator attended about one quarter or less of the time. Survey responses revealed 
that many teachers were not observed by their principals while implementing LDC in the 
classroom; 38% of the teachers reported that their administrator never observed them teach 
an LDC mini-task, 22% reported that they were observed once, and 40% reported being 
observed two or more times. 


Teachers were also asked to rate the support LDC received from school administrators 
using a 4-point agreement scale ranging from strongly disagree to strongly agree (see 
Figure 3.6). Teachers generally agreed or strongly agreed that their school administrators 
supported LDC. For example, more than 80% of teachers indicated that their administrator had 
a firm understanding of LDC, allocated resources, encouraged teacher participation, and 
communicated how LDC supports specific school initiatives and/or goals. Most teachers (62%) 
also disagreed or strongly disagreed that their principal expressed concerns about LDC taking 
time away from other instructional priorities. This matches well with the view of most 
respondents that the LDC program aligned with school initiatives and/or goals. 
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Figure 3.6 
Teacher Perceptions of School Administrator Support 
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Teacher Leadership Role (T37). Next teachers were asked to report on their own roles as 
LDC leaders at their schools. As can be seen in Figure 3.7, more than half of the teachers agreed 
or strongly agreed with each statement. The most commonly reported role (70%) involved 
helping to set instructional goals for the LDC work at their school. About two thirds of teachers 
also reported working with their teacher leader and administrator to shape implementation. 
The least reported role involved wanting to learn more about leading LDC implementation via 
virtual coaching, providing feedback, and so forth. 


Figure 3.7 
Role of Teachers in the Leadership of LDC at Their School 


Teacher Leadership Role (Teacher Survey) 
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LDC Impact (T30-T32) 


This domain covers questions on LDC impact on teacher practice and learning (T30-31) 
and student learning (T32). 


Impact on Teacher Practice and Learning (T30-T31). Almost all respondents reported 
that their teaching skills improved during the 2018-2019 school year, and in most cases, this 
reflected moderate to a great deal of change (see Figure 3.8). The skills with the most ratings 
reporting a great deal of impact included selecting focus standards for a writing assignment, 
creating standards-driven writing assignments, and identifying the skills students need to 
develop to complete a writing assignment. 
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Figure 3.8 


Teacher Perceptions of the Impact of LDC on Their Teaching Practice and Learning 


Impact on Teacher Skills (Teacher Survey) 
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Teachers were also asked their perceptions of the impact of LDC on a ranger of broader 
teacher practices using a 4-point agreement scale ranging from strongly disagree to strongly 
agree. As can be seen in Figure 3.9, the majority of teachers generally agreed or strongly agreed 
with the different impacts. More specifically, the vast majority of teachers agreed that LDC 
raised their expectations for students’ writing (89%), and helped them learn how to incorporate 
writing assignments into their curriculum (88%). Furthermore, while 87% of teachers reported 
that they were now more likely to collaborate when designing instruction, only 59% noted that 
they shared their LDC work with colleagues who were not in their LDC PLC. 


Figure 3.9 
Teacher Perceptions of the Impact of LDC on Their Teaching Practice and Learning 


Impact on Teacher Practice and Learning (Teacher Survey) 
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Impact on Student Learning (T32) 


Teachers were asked about the impact of LDC on their students using a 4-point scale 
ranging from not at all to a great deal (see Figure 3.10). Overall, very few of the teachers 
reported that their students received no positive effect in regards to any of the potential 
impacts listed (1% to 3%). The positive effects that teachers were most likely to endorse 
focused on writing and the obtaining of content knowledge. In contrast, teachers were more 
likely to say that their students only benefitted a little (18% to 25%) with regard to performance 
on assessments and other skills, such as reading, speaking and listening, and college and career 
readiness. 


Figure 3.10 

Teacher Perceptions of the Impact of LDC on Students 
Impact on Students (Teacher Survey) 
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Facilitators and Barriers (T38) 


Teachers were asked to report on potential facilitators or barriers to the implementation 
of LDC (see Figure 3.11). Generally, the findings were positive. More specifically, more than 80% 
indicated that they had sufficient time to meet in their PLC, felt adequately prepared to 
implement modules, had adequate technology to support the use of LDC, and had an easy time 
finding and adapting mini-tasks to implement. More than half of the teachers, however, 
reported that it was challenging to find content-rich reading materials to support their 
modules. 


38 


Figure 3.11 


Teacher Perceptions of Facilitators and Barriers to LDC Implementation 


Facilitators and Barriers (Teacher Survey) 
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3.2 Teacher Leader Survey Results 


This section presents the survey results from teacher leaders. At each school, the teacher 
leader coordinated PLC meetings with LDC coaches. Teacher leaders were either classroom 
teachers or coaches/coordinators. A total of 37 teacher leaders responded to the survey. 


LDC Participation (T1-2) 


Only three of the 37 teacher leaders (8%) who responded to the survey reported that 
2018-2019 was their first year of implementing LDC. The remaining teacher leaders (92%) all 
had 1 or 2 years of previous experience with the program. Among those who were returning to 
LDC, a great deal of variation was reported about module implementation during the 
2016-2017 and/or 2017-2018 school years. More specifically, teacher leaders reported 
previously implementing about two modules (MM = 2.38, range = 0 to 10) and three mini-tasks 
outside of the context of a module (M = 3.06, range = 0 to 16). 


Professional Learning Community and Teacher Collaboration (TL3-TL6) 


Most of the teacher leaders reported that their PLC met every other week (65%) or once a 
month (22%). The most common reasons stated for not meeting every week included PLC 
members having other priorities outside of LDC (46%), limited interest in attending meetings 
(27%), and PLC time not being protected (21%). According to teacher leaders, more informal 
conversations about LDC outside of the PLC meetings typically took place once a week to once a 
month (84%). More than half of respondents (60%) stated that PLC meetings typically lasted 
45-59 minutes, with another third (32%) stating that they lasted 60-74 minutes. 
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Teacher Training and Support (TL7-TL12) 


The vast majority of teacher leaders (87% to 89%) who responded stated that their PLC 
was moderately to very effective in all of the ways listed on the survey. This included creating a 
comfortable working environment, fostering an environment for sharing instructional plans, 
helping teachers improve their LDC instructional plans, and allowing space sharing of student 
work (see Figure 3.12). 


Figure 3.12 
Teacher Leader Perceptions About PLC Effectiveness 
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Teacher leaders were asked their opinions of the online course materials in the LEARN tab 
in LDC CoreTools (see Figure 3.13). Overall, more than two thirds of the teacher leaders rated 
the materials as good to excellent. The most uniformly positive reactions were regarding the 
relevance of the information presented, the clarity of the information, and the opportunity to 
extend learning. Teacher leaders were somewhat less likely to highly rate the usefulness of 
videos and the ease of use of the site. 
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Figure 3.13 

Teacher Leader Perceptions of the Online Course Materials in the LEARN Tab in LDC CoreTools 
Online Course Materials (Teacher Leader Survey) 
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Teacher leaders were also asked to report on the supports they received from their LDC 
coaches. When asked in general, almost all teacher leaders reported that they were able to get 
feedback and support from their LDC coach (97%) either through CoreTools or in the PLC, 
including the receipt of written feedback in a timely manner (97%) through CoreTools. Despite 
this, 8% reported that they did not receive written feedback in LDC CoreTools from their coach. 
Of those who did report receiving the different supports, views were positive with 80% or more 
of teacher leaders reporting them as moderately to very helpful (see Figure 3.14). These 
included written feedback, Zoom video conferences, email or phone communication, or other 
sources of feedback. 


Teacher leaders varied in the amount of in-person and online professional development 
offerings that they received during the 2018-2019 school year. More specifically, while teacher 
leaders reported an average of participating in six professional development offerings, the 
actual numbers reported ranged from a low of zero to a high of 20. 


Figure 3.14 
Teacher Leader Perceptions of the Different Types of LDC Coaching Supports 
Helpfulness of LDC Coach Supports (Teacher Leader Survey) 
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Teacher leaders were asked seven positively oriented items and one negatively oriented 
item concerning the support they received from their LDC coach (see Figure 3.15). As would be 
hoped for, almost all teacher leaders (94% to 100%) agreed or strongly agreed to the positively 
oriented statements. In contrast, only 60% of teacher leaders disagreed or strongly disagreed 
that it was challenging to coordinate with their LDC coach on how to structure PLC time. 
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Figure 3.15 
Teacher Leader Perceptions of the Support They Received From the LDC Coaches 


Support Received From LDC Coach (Teacher Leader Survey) 
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Module Creation (TL13-TL16) 


Teacher leaders generally adapted twice as many modules (M = 2.92, range = 0 - 12) as 
they created (M = 1.38, range = 0 - 6). New modules were most likely to be created by teams of 
two or more teachers (60%), although many reported creating modules individually (46%) or in 
their PLC as a whole (27%). As can be seen in Figure 3.16, almost all teachers felt successful 
with regard to each of the module creation tasks. Furthermore, more than 80% of all 
respondents agreed to a great extent that they were able to select focus standards for a writing 
assignment, create a standards-driven writing assignment, and/or identify the skills that 
students need to develop to complete a writing assignment. Teacher leaders were a little bit 
less likely to be confident in their ability to differentiate instruction, plan for student 
assessment, and assess student progress. 


43 


Figure 3.16 


Teacher Leader Reports on Module Creation Activities 
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Impact on Student Learning (TL17) 


Teacher leaders were asked about the impact of LDC on student learning using a 4-point 
scale ranging from not at all to a great deal (see Figure 3.17). Overall, teacher leaders felt that 
students benefited from LDC. The only two items on which anyone reported not at all included 
reading skills (n = 1) and performance on assessments throughout the school year (n = 1). In 
addition, moderate percentages of teachers reported that their students only benefited a little 
with regard to acquiring college and career skills (27%), students’ capacity to analyze and 
understand the components of a writing assignment (19%), and speaking and listening skills 
(19%). 
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Figure 3.17 

Teacher Leader Perceptions of the Impacts of LDC on Student Learning 
Impact on Student Learning (Teacher Leader Survey) 
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Alignment (TL21) 


Teacher leaders were also asked to rate their agreement about the alignment of LDC with 
the goals and initiatives at their schools (see Figure 3.18). In this case, two thirds or more of the 
teacher leaders agreed or strongly agreed with the seven positively framed statements. For 
example, the vast majority agreed that LDC helped prepare students for current state 
assessments (95%), was a strategy to implement the state standards (92%), and helped 
teachers create writing assignments within their curricula (89%). In addition, 68% of teacher 
leaders agreed or strongly agreed that LDC implementation was connected to the schoolwide 
goals. Finally, only about half of teacher leaders agreed with the negatively framed item, that is, 
that time implementing LDC interfered with other important initiatives at the school. 
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Figure 3.18 
Teacher Leader Perceptions of the Alignment Between the School and LDC 


Alignment (Teacher Leader Survey) 
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Teacher Leader Leadership Role (TL20) 


Next, teacher leaders were asked to report on their own roles as LDC leaders at their 
schools. As can be seen in Figure 3.19, more than three fifths of the teacher leaders (65% to 
92%) agreed or strongly agreed with each statement. The most commonly reported roles 
involved building capacity as an instructional leader (92%) and differentiating LDC 
implementation to meet the learning needs of teachers (89%). The least reported roles involved 
meeting regularly with the school administrator to make planning decisions around LDC (65%) 
and being able to effectively advocate for additional resources on campus (78%). Finally, the 
only item on which someone strongly disagreed (n = 1) involved being able to lead the PLC 
without the assistance of an LDC coach. 
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Figure 3.19 


Leadership Roles of the Teacher Leaders 


Leadership Role (Teacher Leader Survey) 
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Leadership Support (TL18-19, T23) 


This domain covers questions on school administrator support (TL18-19) and district 
support (TL23). 


School Administrator Support (TL18-TL19). According to teacher leaders, administrators 
at many of the schools were engaged with the PLC. More specifically, 41% reported that their 
administrator attended the PLC meetings about three quarters or more of the time, 24% 
reported that their administrator attended about half of the meetings, and 35% reported that 
their administrator attended one quarter or less of the time. 


Next, teacher leaders were asked to rate the level of support that their school 
administrators provide to LDC (see Figure 3.20). Teacher leaders generally agreed or strongly 
agreed that their school administrators supported LDC. For example, more than 80% of 
respondents indicated that their administrator encouraged teachers to participate (92%), 
helped allocate resources to ensure that the LDC team could meet (84%), and made formative 
assessment a priority at the school (81%). Most teacher leaders (76%) also disagreed or strongly 
disagreed that their principal expressed concerns about LDC taking time away from other 
instructional priorities. This finding aligns well with the perception of most teacher leaders that 
LDC aligned with school initiatives and/or goals. 
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Figure 3.20 


Teacher Leader Perceptions of School Administrator Support 
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0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100% 
had a firm understanding of LDC. 
allocated resources...to ensure the LDC team could meet. 


encouraged teachers to participate in LDC. 


expressed concerns that implementing LDC is taking time 
away from other instructional priorities. 
communicated how using LDC’s tools supported specific 
school initiatives and/or goals. 
provided me with feedback about my LDC planning 
and/or instruction. 


made formative assessment a priority at my school. 


Used LDC to implement standards-driven assignments 
within existing curriculum. 


@ Strongly disagree to disagree ™@ Agree to strongly agree 
Note. n= 37. 


District Support (TL23). Overall, teacher leaders had positive attitudes regarding district 
support towards LDC (see Figure 3.21). More specifically, 86% of respondents agreed or 
strongly agreed that district leaders supported LDC implementation. In addition, about 70% of 
respondents agreed or strongly agreed that district leaders had a firm understanding of LDC 
and were interested in expanding its use to other schools, and that LDC aligned with district 
professional development efforts. Finally, 55% reported that district leaders visited the school 
to discuss implementation. 


Figure 3.21 
Teacher Leader Perceptions of District Support of LDC 
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Scale-up and Sustainability (TL22) 


Teacher leaders showed mixed levels of agreement to five questions that focused on 
issues of scale-up and sustainability. Among the most positive results, more than half of 
respondents indicated that they expected most teachers to continue implementing LDC the 
next year (62%), believed that teachers and administrators were committed to sustaining LDC 
(62%), and felt that LDC led to new collaborations across grades and/or subjects (54%). In 
contrast, only 38% of respondents agreed or strongly agreed that the LDC planning process 
and/or CoreTools was used by teachers at their schools who were not part of the PLC meetings. 
Likewise, only 32% believed that their LDC PLC would increase in size during the next school 
year (see Figure 3.22). 


Figure 3.22 
Teacher Leader Perceptions of the Scale-Up and Sustainability of LDC 


Scale-up and Sustainability (Teacher Leader Survey) 
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3.3 Administrator Survey Results 


In total, 35 school administrators spanning 29 schools (out of 34 total participating in LDC 
in 2018-2019) completed the survey. This included 23 principals, eight assistant principals, and 
four others (e.g., counselor, coordinators, and a principal designee). 


Professional Learning Community (A2) 


Administrators varied in the proportion of the PLC meetings that they attended. More 
specifically, 40% reported that they attended about three quarters or more of the meetings. In 
addition, 23% reported that they attended half of the meetings and another 20% reported 
attending about one quarter. Finally, 17% indicated that they attended less than one-quarter of 
the PLC meetings. It is not clear from the question whether any of the administrators failed to 
attend at all. 


Training and Support (A3-A4) 


When asked about issues of training and support, all administrators who responded to the 
survey agreed or strongly agreed to each statement (see Figure 3.23). The strongest levels of 
agreement focused on whether the administrators were able to reach LDC staff when they had 
questions (83%), there was adequate technology to access online resources (83%), and LDC 
staff were able to connect with them when they needed additional resources (77%). Finally, 
more than two-thirds reported that they and their teacher leaders each had sufficient PD 
opportunities (71% and 69%, respectively). Furthermore, while administrators attended an 
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average of five in person and/or online professional development sessions, some participated a 
great deal more or less (1 to 20). 


Figure 3.23 
Administrator Perceptions of Training and Support 


Training and Support (Administrator Survey) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


| was able to reach LDC staff when | had questions about 
LDC. 


My school has adequate technology to access LDC online 
resources. 


LDC offered sufficient professional development 
opportunities for LDC teacher leaders. 


LDC offered sufficient professional development 
opportunities for school administrators. 


LDC staff members were able to connect me with 
additional resources when needed. 


MAgree M™Strongly agree 


Note. n= 35. 


Classroom Observation (A5-6) 


Thirty-three of the 35 administrators who responded to the survey indicated that they 
observed each member of their PLC teaching an LDC module. Overall, the most common 
response among administrators was three or more observations per teacher (40%), with about 
one third only observing once each (34%), and the remaining reporting observing each teacher 
twice (20%). In addition, administrators were asked how effective they thought the modules 
they observed were at helping students develop literacy skills. Almost all perceived that the 
modules were moderately (46%) or very effective (48%), with only two administrators (6%) 
deeming it to be only a little effective. None of the administrators reported that the modules 
were not effective in helping develop students’ literacy skills. 


Impact on Teacher Practice (A7) 


Administrators were asked their perceptions about the level and types of impact that LDC 
had on teacher practice at their PLC (Figure 3.24). Overall, perceptions were positive with 80% 
to 96% of administrators indicating that LDC had a moderate to a great deal of impact on 
teachers, depending on the practice. Administrators were most likely to report a positive 
impact on teachers’ ability to identify patterns of student understandings or misconceptions 
(96%), ability to create standards-driven writing assignments (94%), and ability to select focus 
standards and identify skills for writing assignments (91%, respectively). Administrators were 
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somewhat less likely to perceive an impact on teachers’ ability to collect information 
systematically on students’ progress (80%). 


Figure 3.24 


Administrator Perceptions of the Impact of LDC on Teacher Practice 


Impact on Teacher Practice (Administrator Survey) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Selecting a set of focus standards for a writing 
assignment. 


Creating standards-driven writing assignments. 


Identifying the skills students need to develop to 
complete a writing assignment. 

Creating daily lessons to teach each skill a student needs 
to complete a writing assignment. 

Systematically collecting information on students’ 

progress. 
Identifying patterns of student understandings or 
misconceptions. 

Using evidence of student progress on standards to 

modify subsequent instruction 


HNotatalltoalittle |§™|Moderately toa great deal 


Note. n= 35. 


Impact on Student Learning (A8) 


Administrators also perceived a variety of positive impacts on the students whose 
teachers implemented LDC (Figure 3.25). For example, most indicated that LDC had a moderate 
to a great deal of impact on students’ writing including completing assignments (97%), the 
quality of the work (91%), and their capacity to analyze and understand the components of a 
writing assignment (89%). The areas where administrators were the least likely to perceive 
positive impacts (71%) included reading skills, speaking and listening skills, and/or performance 
on assessments throughout the school year. 


Figure 3.25 
Administrator Perceptions of the Impact of LDC on Student Learning 


Impact on Student Learning (Administrator Survey) 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Reading skills 
Content knowledge 
Ability to complete writing assignments 
Quality of students’ writing 
College and career ready skills 
Capacity to analyze and understand the components of... 
Speaking and listening skills 


Overall literacy performance 


Performance on assessments throughout the school year 


HNotatalltoalittle © ™|Moderately toa great deal 


Note. n= 35. 


Administrator Leadership Role (A9) 


Next, administrators were asked to report on their own roles as LDC leaders at their 
schools. As can be seen in Figure 3.26, more than three fourths of the administrators who 
responded (77% to 91%) agreed or strongly agreed with each statement. The most commonly 
reported roles involved meeting with the teacher leader to stay abreast of implementation 
progress, participating in discussions about how to differentiate LDC implementation, and 
allocating resources (91% in each case). In addition, 86% agreed or strongly agreed that they 
were able to shape LDC implementation at their school and/or make changes to school 
schedules to accommodate LDC professional learning time. Finally, the least common role 
(77%) involved discussing how to expand implementation during future school years. 


53 


Figure 3.26 
Administrator Perceptions of Their Leadership Role With LDC 


Administrator Leadership Role (Administrator Survey) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


| was able to shape LDC implementation at my school. 


| met regularly with the LDC teacher leader in my school 
to stay abreast of implementation progress. 


| was involved in discussions about differentiating LDC 
implementation to meet teacher learning needs. 
| led discussions about how to expand my school’s LDC 
implementation in future years. 
| made changes to school schedules to accommodate LDC 
professional learning time. 


| allocated resources such as teacher time, payment, 
administrator time, support staff, sub coverage, etc., to... 


@ Strongly disagree to disagree ™@ Agree to strongly agree 


Note. n= 35. 


Alignment (A10) 


Administrators were also asked to rate their agreement about the alignment of LDC with 
the goals and initiatives at their schools (see Figure 3.27). Two thirds or more of the 
administrators agreed or strongly agreed with the seven positively framed statements. For 
example, the vast majority agreed that LDC complemented other initiatives at their schools 
(97%), was a strategy for implementing the state standards (95%), helped teachers create 
writing assignments within their curricula (94%), and prepared students for state assessments 
(91%). Finally, less than half of the administrators agreed with the two negatively framed items, 
that is, that time implementing LDC interfered with other important initiatives at the school 
(37%) and that it was difficult to focus on LDC because of other school priorities (43%). 


Figure 3.27 
Administrator Perceptions of LDC Alignment With the School 


Alignment (Administrator Survey) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100% 


Our school connected LDC implementation to our specific 
schoolwide goals. 
LDC helped teachers create writing assignments to use 
within their current curricula. 
LDC complemented other initiatives taking place in my 
school. 
| view LDC as a strategy for implementing my state’s 
College- and Career-Ready Standards. 
The time spent implementing LDC interfered with other 
important initiatives at my school. 
LDC helped prepare students in my school for current 
state assessments. 
It was difficult for teachers to focus on LDC because of 
other competing priorities at the school. 
| am using LDC to implement standards-driven 
assignments within the existing curriculum. 


LDC helped improve teacher evaluation ratings. 


Strongly disagree to disagree ™@ Agree to strongly agree 


Note. n= 35. 


Scale-up and Sustainability (A11) 


As can be seen in Figure 3.28, administrators showed mixed levels of agreement to the 
five questions that focused on issues of scale-up and sustainability. Among the most positive 
results, more than two thirds of respondents indicated that they expected most teachers to 
continue implementing LDC the next year (91%), believed that teachers and administrators 
were committed to sustaining LDC (88%), and felt that LDC led to new collaborations across 
grades and/or subjects (73%). In contrast, only 31% of respondents agreed or strongly agreed 
that the LDC planning process and/or CoreTools was used by teachers at their schools who 
were not part of the PLC meetings. Finally, only 48% believed that their LDC PLC would increase 
in size during the next school year. 
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Figure 3.28 
Administrator Perceptions of the Scale-up and Sustainability of LDC 


Scale-up and Sustainability (Administrator Survey) 
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As a result of LDC, new collaborations across grades 0 —SCiBH 
and/or subjects were created or are being launched at... 
Teachers and administrators at my school are committed =U 88% 
to sustaining the LDC initiative. (n = 34) 
| expect our LDC PLC to increase in size next year. (n = 33) ees 48% 


™@ Strongly disagree to disagree ™@ Agree to strongly agree 


District Support (A12) 


Overall, administrators had positive attitudes regarding district support towards LDC (see 
Figure 3.29). More specifically, 81% of respondents agreed or strongly agreed that district 
leaders supported LDC implementation. In addition, a little over two thirds agreed or strongly 
agreed that district leaders were interested in expanding its use to other schools (71%) and/or 
that district leaders had a firm understanding of LDC (68%). Finally, 62% felt that district 
professional development efforts were aligned with LDC and 52% indicated that district leaders 
visited the school to discuss LDC implementation. 
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Figure 3.29 
Administrator Perceptions of District Support of LDC 


District Support (Administrator Survey) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


District leaders supported the implementation of LDC. (n 
= 26) 


District leaders had a firm understanding of LDC. (n = 25) 


District leaders are interested in spreading the use of LDC 
to additional schools. (n = 21) 
District professional development efforts were aligned 
with the LDC initiative. (n = 29) 
District leaders visited my school to discuss the 
implementation of LDC. (n= 31) 


™@ Strongly disagree to disagree @ Agree to strongly agree 


3.4 Open-Ended Responses for All Participants 


Each survey respondent was asked three open-ended questions regarding the supports 
they received for the implementation of LDC at their school: (a) What supports did you find the 
most useful and why? (b) What supports were not helpful and why? and (c) In what ways could 
LDC implementation be improved in your school in the future? The following section 
summarizes the findings for the 192 teachers, 37 teacher leaders, and 36 administrators who 
responded to the open-ended questions. 


Results were analyzed thematically for the supports of coaching, CoreTools, and 
professional development, all of which were mentioned in the survey question prompt. In 
addition, we coded responses for collaboration and other more general issues that did not 
clearly fit within the other major themes (for example, responses that mentioned meetings but 
did not explicitly state whether they were for coaching, collaboration, or professional 
development). Finally, we coded responses that did not answer the question: for example, 
responses that said not applicable or nothing, people who only made negative comments when 
asked to say what was most useful, and individuals who only said something positive in regards 
to the question about what was least helpful. A summary of the coding can be found in Table 
3,2. 
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Table 3.2 
Major Themes From Open-Ended Survey Items About LDC Supports 


Teacher Teacher leader Administrator Total 
Supports # % # % # % # % 

Most useful 

Collaboration 33 17.2 2 5.4 2 5.6 37 14.0 
Coaching 121 63.0 26 70.3 29 80.6 176 66.4 
CoreTools 73 38.0 22 59.5 17 47.2 112 42.3 
Professional development 29 15.1 10 27.0 6 16.7 45 17.0 
Other 11 5.7 7 18.9 0 0.0 18 6.8 
Least useful 

Collaboration 2 1.0 0 0.0 0 0.0 2 0.8 
Coaching 47 24.5 4 10.8 3 8.3 54 20.4 
CoreTools 42 21.9 12 32.4 7 19.4 61 23.0 
Professional development 19 9.9 2 5.4 2 5.6 23 8.7 
Other 19 9.9 2 5.4 3 8.3 24 9.1 
Suggestions 

Collaboration 22 11.5 2 5.4 5 13.9 29 10.9 
Coaching 25 13.0 5 13.5 7 19.4 37 14.0 
CoreTools 30 15.6 6 16.2 2 5.6 38 14.3 
Professional development 16 8.3 3 8.1 3 8.1 22 8.3 
Other 112 58.3 24 64.9 20 55.6 156 58.9 


Note. Each comment was coded for one or more themes; therefore, the total percentages could exceed 
100%. 


General Views 


About one third of the responses to the open-ended questions did not directly respond to 
the question being asked. For example, 93 of the responses across the three questions had a 
neutral statement such as not applicable or nothing. In addition, 60 of the responses to the 
question about what was least helpful and nine of the responses to the question asking for 
suggestions simply provided a positive comment such as noting that the supports were helpful. 
Only seven responses across the three questions made a negative statement such as their not 
being interested in participating again. It should also be noted that eight of the respondents to 
the question about what support was most helpful put a general positive statement indicating 
that they found many or all of the supports listed to be helpful. 
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Collaboration 


While not mentioned in the section prompt, a moderate number of respondents brought 
up collaboration as a form of support that they considered helpful (37 comments) or that they 
had suggestions about for the future (29 comments). Specific themes included the value of 
getting to engage with their colleagues to not only build community, but also to share 
materials, develop exemplars of student work, or plan or revise modules to name just a few. A 
number of respondents also highlighted the opportunity to work with or meet in grade level or 
department groupings. With regard to suggestions, the most salient theme involved desiring 
greater time to collaborate on the development, adaptation, or analysis of results for the 
modules. Furthermore, four respondents specifically mentioned wanting more PLC meeting 
time for collaboration, and seven respondents wanted more time to work in grade-level or 
department-level groups. Finally, in response to the question about what was not helpful, one 
teacher indicated that they wanted more department meetings to collaborate, and another 
noted that they never had a chance to collaborate at their school. 


Coaching 


Based on the open-ended responses, coaching was the most valued type of support 
provided for the implementation of LDC (176 comments). The majority of these comments 
focused on one or more of the modes in which the coaching was provided. More specifically, 33 
respondents valued the in-person coaching, 78 valued the virtual coaching, and 22 valued 
coaching through both of these modes. In some cases, respondents provided extra detail, with 
52 noting that they valued using Zoom for the virtual coaching, six mentioning the use of email, 
and one noting that they appreciated the coaching via CoreTools. A moderate number of 
respondents to this question also mentioned the person who provided the coaching for their 
school (68 comments). For example, respondents mentioned their LDC coordinator, LDC coach, 
teacher leader, district lead, and/or provided the name of an individual who provided coaching. 
Finally, 34 respondents pointed out that one of the benefits of the coaching was the 
opportunity to receive written or verbal feedback. 


Participants also provided comments about what they felt did not work well or what they 
would like to have seen regarding the coaching. The most common theme regarding what did 
not work well involved the virtual coaching (42 comments). Individuals particularly felt that 
Zoom was problematic (29 comments) with some pointing out that they had technology issues 
and/or that it affected engagement across the group. While seven participants also mentioned 
the in-person coaching, these comments tended to focus on the lack of these opportunities, 
especially outside of the confines of the PLC meetings. The results for the question about 
suggestions align with these results. That is, 19 individuals commented that they would like 
more in-person coaching, seven indicated that they would like less virtual coaching, and only 
five said that they wanted more virtual opportunities. Finally, a few respondents made 
suggestions about content for the coaching, such as greater support in choosing modules to 
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adapt, modeling lessons, and providing feedback about how teachers can improve their 
modules. 


CoreTools Online Platform 


Based on their comments, teachers, teacher leaders, and administrators all felt that there 
were positive (112 comments) and less positive (61 comments) aspects to the CoreTools 
platform. While some of the respondents who liked CoreTools did not specify why it was a 
helpful support, those who did were most likely to highlight the module creator (16 comments), 
the LEARN tab that includes various professional development materials (19 comments), or the 
library (40 comments) that includes different mini-tasks and modules that can be integrated or 
adapted. For those respondents who felt that there were drawbacks to CoreTools, the most 
cited issues involved the library (11 comments), the ease of use of the platform (18 comments), 
and the LEARN tab (27 comments). Suggestions that respondents made focused on similar 
themes including the LEARN tab (four comments), the module creator (nine comments), library 
(10 comments), and/or ease of use (11 comments). Among the suggestions made in response 
to both questions were to make the navigation of CoreTools and the LEARN tab easier and 
more intuitive, improve the breadth of modules or mini-tasks in the library (e.g., grades, 
content areas, English learners, etc.), and improve the ability for teachers and/or administrators 
to collaboratively work in the module creator. 


Professional Development 


Respondents also pointed out more positives (45 comments) than negatives (23 
comments) regarding the professional development they were offered as part of their 
implementation of LDC. More specifically, moderate numbers of respondents indicated that 
they found the in-person professional development (14 comments) and/or the professional 
development opportunities that took place on the weekend or during the summer (13 
comments) the most helpful. In addition, while 11 respondents indicated that they found the 
LEARN tab course(s) to be a helpful form of professional development, 14 respondents did not 
like one or more aspects of the online courses and an additional two respondents did not do 
any online courses. Finally, small numbers of respondents noted that they found the 
professional development provided by their district (two comments) or the i3 grant director 
(two comments) to be helpful, and another respondent suggested that the district provide the 
initial in-person training. 


Other General Views 


As previously noted, some responses did not clearly align with the four main themes. 
While these were mostly found in response to the question asking for suggestions, some were 
found for the other two questions. The most common of these were focused on issues of time 
and/or meetings with four individuals making positive statements and 15 pointing out issues 
such as needing more time. Time was also a very common theme among the suggestions (79 
comments) with most respondents wanting more time for meetings, planning, and/or 
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implementation of the modules. A moderate number of respondents also suggested that their 
school expand participation (44 comments) by including more teachers, grade levels, and/or 
subject areas. As one teacher stated, “Involve the lower grades so there is more continuity.” 
Finally, 14 respondents indicated that they would like to see more buy-in from administrators 
at the school and/or district level and another 14 noted that they would like to see more 
alignment in regards to competing initiatives, pacing plans, curricula, etc. 


3.5 Exploratory Analysis 


In order to examine change in the perceived impacts among returning teachers, survey 
ratings were compared for teachers who submitted a survey during both spring 2018 and spring 
2019. The following presents the descriptive results for the 121 teachers who participated at 
both time points. 


When examining the longitudinal results regarding the impact of LDC on their teaching 
skills, the percentage of teachers who provided a positive rating of moderately to a great deal 
increased across the 2 school years (see Figure 3.30). More specifically, percentages increased 
by 8% to 13% for each item. Furthermore, by spring 2019, the percentage of teachers providing 
positive ratings met or exceeded 90% on all but one item. In addition, as can be seen in Table 
3.3, when looking at the direction of change in ratings across the 4-point scale (not at all toa 
great deal), most teachers ratings showed no change (45.5% to 56.2%) or increased by one or 
more levels (29.8% to 37.2%). 
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Figure 3.30 
Change in Teacher Perceptions of the Positive Impact of LDC on Teacher Skills 


Change in Positive Impact on Teacher Skills (Teacher Survey) 
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Table 3.3 
Change in Teacher Perceptions of the Impact of LDC on Teacher Skills (n = 121) 


Negative No change Positive 

Dimension # % # % # % 
Selecting focus standards for a writing assignment. 17 14.0 68 56.2 36 29.8 
Creating standards-driven writing assignments. 14 11.6 62 51.2 45 37.2 
Identifying the skills students need to complete a 19 15.7 65 53.7 37 30.6 
writing assignment. 
Creating daily lessons to teach the skills students 19 15.7 55 45.5 47 38.8 
need to complete a writing assignment. 
Systematically collecting information on students' 20 16.5 57 47.1 44 36.4 
progress. 
Identifying patterns of student understandings or 21 17.4 56 46.3 44 36.4 
misconceptions. 
Using evidence of student progress onstandardsto 20 16.5 62 51.2 39 32.2 


modify subsequent instruction. 


Note. Four-point agreement scale ranging from not at all to a great deal. 


Positive growth was also found when examining change in teachers’ perceptions about 
the impact of LDC on various facets of their teaching practice, including collaboration and 
standards alignment (see Figure 3.31). First, percentages of teachers who gave a positive rating 
(i.e., agree or strongly agree) increased for each of these items. In this case, though, there was 
more variation in the positive change (3% to 16%). Furthermore, the largest amount of growth 
was found for improving on their teacher evaluation ratings as well as sharing work with 
colleagues outside of the PLC. Positive change in the perceived impact of LDC on teacher 
evaluation ratings and the propensity of teachers to share work with colleagues was particularly 
marked. As shown in Table 3.4, most teachers’ ratings were stable (44.6% to 60.3%) or 
increased by one or more levels on the agreement scale (26.4% to 39.7%). 


Figure 3.31 


Change in Teacher Perceptions of the Positive Impact of LDC on Their Teaching Practice and Learning 


Change in Positive Impact on Teacher Practice and Learning (Teacher Survey) 
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LDC helped me incorporate writing assignments into my 
existing curriculum. 


lam more likely to collaborate with other teachers on 
designing instruction after participating in our LDC PLC. 


LDC helped me improve on my teacher evaluation 
ratings. 
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Table 3.4 


Change in Teacher Perceptions of the Impact of LDC on Teacher Practice and Learning, Counts and 
Percentages (n = 121) 


Negative No change Positive 

Dimension # % # % # % 
Participating in LDC raised my expectations for 15 12.4 70 57.9 36 29.8 
students’ writing. 
Using LDC modules became an important part of 15 12.4 65 53.7 41 33.9 
my instructional practice. 
Implementing LDC helped me incorporate my 16 13.2 73 60.3 32 26.4 
state’s CCR standards into my instruction. 
LDC helped me incorporate writing assignments 20 16.5 65 53.7 36 29.8 
into my existing curriculum. 
lam more likely to collaborate with other teachers 24 19.8 63 52.1 34 28.1 
on designing instruction after participating in our 
LDC PLC. 
LDC helped me improve on my teacher evaluation 17 14.0 56 46.3 48 39.7 
ratings. 
Participating in LDC helped me develop working 19 15.7 68 56.2 34 28.1 
relationships with teachers in different grades 
and/or subjects. 
| shared my LDC work with colleagues outside of 21 17.4 54 44.6 46 38.0 


the LDC PLC. 


Note. Four-point agreement scale ranging from strongly disagree to strongly agree. 


Finally, we examined changes in teacher ratings concerning the impact of LDC on nine 
different student outcomes. As can be seen in Figure 3.32, there were small to moderate 
increases (1% to 12%) in the percentage of teachers who gave positive ratings (moderately to a 
great deal) for each of the items. The largest growth in positive ratings was found for student 
improvement in content knowledge, reading skills, and the quality of students’ writing. Despite 
this last point, there was only a 1% increase in positive ratings concerning students’ ability to 
complete writing assignments. Finally, results concerning general changes in ratings mirrored 
those found for the other two longitudinal analyses (see Table 3.5). More specifically, the 
largest proportion of teachers showed no change in their ratings for the items between the two 
survey administrations (46.3% to 61.2%), with the next largest proportion showing an increase 
of one or more levels on the agreement scale per item (21.5% to 38.0%). 
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Figure 3.32 
Change in Teacher Perceptions of the Positive Impact of LDC on Students 


Change in Positive Impact on Students (Teacher Survey) 
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Table 3.5 
Change in Teacher Perceptions of the Impact of LDC on Students, Counts and Percentages (n = 121) 


Negative No change Positive 
Dimension # % # % # % 

Reading skills 20 16.5 61 50.4 40 33.1 
Content knowledge 20 16.5 62 51.2 39 32.2 
Ability to complete writing assignments 21 17.4 74 61.2 26 21.5 
Quality of students’ writing 21 17.4 59 48.8 41 33.9 
College and career ready skills 23 19.0 60 49.6 38 31.4 
Capacity to analyze and understand the 21 17.4 63 52.1 37 30.6 
components of a writing assignment 

Speaking and listening skills 20 16.5 56 46.3 45 37.2 
Overall literacy performance 17 14.0 64 52.9 40 33.1 
Performance on assessments throughout the school 16 13.2 59 48.8 46 38.0 
year 


Note. Four-point agreement scale ranging from not at all to a great deal. 


3.6 Summary of Results 


This survey captured the responses of 87% of all participants in the district’s third year of 
implementation. Based on the survey analyses, a majority of teachers and teacher leaders 
implemented LDC as expected. Most reported implementing multiple modules across the 
school year. The majority of participants also reported that they developed or adapted their 
module(s) collaboratively and that they primarily adapted existing modules. About two thirds of 
participants also noted on their surveys that they found the CoreTools platform and/or 
professional development to be helpful. Despite this, some did report that the CoreTools site in 
general and the LEARN tab specifically could be difficult to navigate, and that the videos were 
not always helpful. In addition, while the teachers who made use of the National Peer Review 
process found it helpful, most teachers reported that they did not make use of this LDC feature. 


Participants tended to feel supported in their implementation of LDC. At the local level, 
teachers had very positive opinions of their teacher leaders, found the opportunity to 
collaborate with their peers to be helpful, and frequently reported that their administrator(s) 
provided encouragement and resources. Teachers and teacher leaders also tended to report 
positive opinions of the LDC coaches, and many noted that the coaching was one of their most 
valued supports. In addition, while teacher leaders and administrators felt that the district 
supported implementation of LDC, moderate percentages felt that district leadership did not 
have a firm understanding of the program, was not interested in spreading its use, or did not 
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align other district professional development with the initiative. Some participants also 
expressed concerns about whether participants in their PLC were committed to sustaining LDC, 
such as recruiting more teachers to participate during the next school year. 


Participants perceived multiple positive impacts of the LDC initiative that aligned with the 
initiative. For example, most teachers, teacher leaders, and administrators felt that the 
program helped students to develop their writing and literacy skills. Teachers and 
administrators felt that teachers improved their ability to select focus writing standards and 
identify the skills that students need to develop to complete writing assignments. In addition, 
most teachers reported improvement in their expectations for their students’ writing, their 
ability to incorporate writing assignments into the existing curriculum, and their likeliness to 
collaborate on the design of instruction. Finally, the areas with the least perceived impact 
aligned less directly with LDC, and included issues such as the development of college and 
career ready skills, speaking and listening skills, and/or performance on assessments. 


Finally, an exploratory analysis was conducted to examine change in teachers’ perceptions 
of the impacts of LDC for the 121 teachers who submitted a survey in both spring 2018 and 
spring 2019. As with the primary analyses for the three scales we examined (teacher skills, 
teacher practice and learning, student skills) we found positive results. More specifically, the 
percentage of teachers who provided positive ratings increased for every item. In addition, 
when examining general change in ratings across all levels of the agreement scales, we found 
that most teachers provided the same or a higher rating. In other words, teachers tended to 
maintain or improve in their positive perceptions about the impact of LDC on their own 
practice, as well as on their students’ skills. 
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4.0 Analyses of LDC CoreTools Data 


In the following section, we report on how participants interacted with LDC’s CoreTools 
online system. We begin by presenting participation rates for key CoreTools activities, including 
creating a user account, and viewing, editing, and commenting on modules. We then dig 
deeper into the three CoreTools activities, by providing descriptive statistics for all i3 CoreTools 
users followed by results broken down by key subgroups (including LDC role, school level, 
cohort, and content area taught). After that, we compare the level of engagement for i3 
CoreTools users who completed and taught a full-length module to those users who did not 
complete and teach a module during the school year. Finally, we summarize results. 


4.1 CoreTools Activity Participation Rates 


The four key CoreTools activities we examined are creating a user account, viewing 
modules, editing modules, and commenting on modules. With the exception of two 
administrators, all participants created CoreTools user accounts. Among the 310 CoreTools 
users, 264 were classroom teachers (with 32 of those individuals acting as teacher leaders), five 
were coaches or coordinators playing the teacher leader role, and 41 of them were 
administrators. 


As seen in Table 4.1, most participants used CoreTools to at least some degree. Ninety- 
nine percent of all participants created a user account, 88% of participants viewed modules, 
and 70% edited modules. Commenting on modules was a less common activity with only 34% 
of all participants doing so. Teachers and teacher leaders also had higher participation rates 
than the principals and assistant principals, as expected. 


Table 4.1 
CoreTools Feature Participation Rates: 2017-2018 (Counts and Percentages) 


User Viewed a Edited a Commented 

Participant type n account module module on a module 
Teacher 232 232 (100%) 219 (94%) 184 (79%) 81 (35%) 
Teacher leader (classroom teacher) 32 32 (100%) 32 (100%) 29 (91%) 20 (63%) 
Total teachers 264 264(100%) 252 (95%) 213 (81%) 101 (38%) 
Teacher leader (coach/coordinator) 5 5 (100%) 4 (80%) 3 (60%) 3 (60%) 
Administrator (principals and 43 41 (95%) 20 (47%) 1 (2%) 2 (5%) 
assistant principals) 
Total participants 312 310 (99%) 275 (88%) 217 (70%) 106 (34%) 
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As previously noted, nearly all participants created a CoreTools user account. Significantly, 
even administrators, who generally were not a part of the regular PLC meetings, 
overwhelmingly created user accounts. This suggests that the vast majority of administrators 
overseeing LDC PLCs had at least some familiarity with the online platform. Likewise, nearly all 
PLC teachers and teacher leaders viewed modules in CoreTools. A little less than half of 
principals and assistant principals viewed modules in the platform. 


We consider editing modules to be the key indicator of deep engagement with the 
CoreTools module building platform. Eighty-one percent of participating teachers edited at 
least one module. In contrast, only one administrator engaged at the level of editing modules. 


Overall, adding comments to modules was a much less common activity. Less than 40% of 
classroom teachers and only five coaches, coordinators, or administrators commented. 
Commenting was, not surprisingly, most common among teacher leaders, whose role was most 
conducive to giving feedback to PLC members. 


4.2 Engagement With Key CoreTools Activities 


In this section, we describe participants’ level of engagement with three key CoreTools 
activities: viewing modules, editing modules, and commenting on modules. Descriptive 
statistics are reported for all participants, as well as a number of subgroups. These subgroups 
capture the role the individual played in LDC (teacher, teacher leader, and administrator), the 
level of the school at which the participant worked (elementary, K—8, middle, 6-12, or high), 
the cohort group describing when the school and individual participant began taking part in 
LDC, and in the case of teachers, the content area taught (elementary/multiple subjects, 
secondary ELA, secondary history/social studies, and secondary science). As noted earlier, some 
participants played multiple roles in the intervention, so the teacher and teacher leader groups 
overlap to some degree as they do in our survey analysis. Descriptive statistics are only 
reported for groups with five or more members. The samples for the viewing, editing, and 
commenting analyses are the 310 LDC participants with CoreTools user accounts in 2018-2019 
(see Table 4.1). 


Module Viewing 


In Table 4.2 we present descriptive statistics on how many times i3 participants viewed 
modules, both overall and by subgroup. We present the minimum and maximum number of 
views, means and standard deviations, and the sum of total views across participants. Overall, 
the average participant viewed modules about 37 times, although the range varied greatly from 
zero to 515 views. There were 35 participants with user accounts who did not view any 
modules (about 11% of the overall sample). 


Differences in module viewing by participant role were dramatic. Teacher leaders viewed 
over twice the number of modules as teachers on average (95.5 compared to 51.0). 
Administrators also viewed modules much less frequently (3.2). Participants in Cohort 2 schools 
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on average viewed more modules than participants in Cohort 1 schools. Participants newly 
joining LDC in 2018-2019 at schools in both cohorts viewed fewer modules than their peers 
who had participated in prior years as well. Participants in K-8 and 6—12 schools viewed more 
modules than their peers in elementary, middle, or high school. Finally, secondary ELA and 
social studies/history teachers viewed more modules on average than secondary science 
teachers and elementary teachers. 


Table 4.2 


Descriptive Statistics for the Number of Times a Participant Viewed a Module in CoreTools, by Participant 
Subgroup 


Subgroup n Min Max M SD Sum 


All participants 310 0 515 36.7 52.2 11,387 


Participant role 


All teachers 264 0 515 41.0 52.1 10,827 
All teacher leaders 37 0 515 95.5 107.9 3,534 
All administrators 41 0 35 3.2 6.8 133 
Cohort 
Cohort 1 / teacher started 2016-2017 42 0 287 23.4 49.4 982 
Cohort 1 / teacher started 2017-2018 16 0 177 34.8 41.3 556 
Cohort 1 / teacher started 2018-2019 21 0 110 17.2 24.3 361 
Cohort 2 / teacher started 2017-2018 149 0 515 44.2 64.1 6,584 
Cohort 2 / teacher started 2018-2019 82 0 132 35.4 30.0 2,904 
School level 
Elementary school participants 202 0 515 35.3 51.1 7,129 
K-8 participants 21 0 291 61.2 80.2 1,285 
Middle school participants 57 0 238 26.6 37.5 1,518 
6-12 participants 21 5 269 54.3 55.6 1,141 
High school participants 9 0 177 34.9 54.4 314 
Content area taught 
Elementary/multiple subjects 177 0 515 40.1 52.4 7,093 
Secondary ELA 36 0 238 50.7 60.1 1,825 
Secondary social studies/history 25 0 269 52.8 67.8 1,321 
Secondary science 17 0 95 29.8 28.4 506 
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Module Editing 


As shown in Table 4.3, on average participants made about 12 edits over the course of the 
school year. There was a wide range of engagement from editing zero times to making 227 
edits to modules. Ninety-three participants with CoreTools user accounts (30%) did not do any 
editing of modules. 


Table 4.3 


Descriptive Statistics for the Number of Times a Participant Edited a Module in CoreTools, by 
Participant Subgroup 


Subgroup n Min Max M SD Sum 
All participants 310 0 227 12.1 22.0 3,760 
Participant role 
All teachers 264 0 227 13.7 22.7 3,623 
All teacher leaders 37 0 227 36.5 46.1 1,352 
All administrators 41 0 1 0.0 0.2 1 
Cohort 
Cohort 1 / teacher started 2016-2017 42 0 91 8.5 16.6 355 
Cohort 1 / teacher started 2017-2018 16 0 57 11.8 16.6 189 
Cohort 1 / teacher started 2018-2019 21 0 15 4.4 5.2 93 
Cohort 2 / teacher started 2017-2018 149 0 227 15.6 27.9 2,329 
Cohort 2 / teacher started 2018-2019 82 0 63 9.7 13.5 145 
School level 
Elementary school participants 202 0 120 10.3 17.5 2,084 
K-8 participants 21 0 291 27.8 51.2 583 
Middle school participants 57 0 63 8.6 14.4 491 
6-12 participants 21 0 117 23.9 25.5 501 
High school participants 9 0 48 11.2 14.8 101 


Content area taught 


Elementary/multiple subjects 177 0 120 11.7 18.0 2,062 
Secondary ELA 36 0 227 20.9 39.1 751 
Secondary social studies/history 25 0 227 26.2 48.4 656 
Secondary science 17 0 58 11.7 16.2 198 
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Subgroup results for the module editing mirrored those found for page viewing. There 
was a dramatic difference between teacher leaders and teachers, with teacher leaders editing 
more than twice the number of modules on average. Administrators simply did not engage at 
this level in 2018-2019, with just one administrator making a single edit. While Cohort 2 
participants edited more modules on average, participants joining LDC in 2018-2019 edited 
fewer modules. Teachers in K-8 and 6—12 schools as well as those teaching secondary ELA and 
social studies/history edited modules at greater levels than their peers. 


Module Commenting 


Only 106 participants, representing about one third of all participants, made at least one 
comment on a module. As shown in Table 4.4, while participants commented between zero and 
42 times, the average was just over two times. Across participant roles, teacher leaders had the 
highest level of engagement, with nine comments on average. 
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Table 4.4 


Descriptive Statistics for the Number of Times a Participant Commented on a Module in 
CoreTools, by Participant Subgroup 


Subgroup n Min Max M SD Sum 
All participants 310 0 42 2.2 6.2 691 
Participant role 
All teachers 264 0 42 2.5 6.5 647 
All teacher leaders 37 0 42 8.6 12.6 317 
All administrators 41 0 6 0.2 0.9 7 
Cohort 
Cohort 1 / teacher started 2016-2017 42 0 38 2.5 6.2 103 
Cohort 1 / teacher started 2017-2018 16 0 21 3.1 5.9 49 
Cohort 1 / teacher started 2018-2019 21 0 7 0.7 1.6 15 
Cohort 2 / teacher started 2017-2018 149 0 42 2.5 7.0 379 
Cohort 2 / teacher started 2018-2019 82 0 38 1.8 5.55 145 
School level 
Elementary school participants 202 0 38 2.1 6.1 429 
K-8 participants 21 0 35 4.3 9.1 90 
Middle school participants 57 0 42 2.3 6.2 130 
6-12 participants 21 0 3 0.3 0.7 6 
High school participants 9 0 21 4.0 7.1 36 
Content area taught 
Elementary/multiple subjects 177 0 38 2.4 6.4 426 
Secondary ELA 36 0 42 4.1 9.6 149 
Secondary social studies/history 25 0 35 3.2 7.4 81 
Secondary science 17 0 8 1.3 2.1 22 


4.3 CoreTools Engagement as an Implementation Variable 


To evaluate the validity of CoreTools engagement as an indicator of LDC implementation, 
we examined the relationship between the three CoreTools engagement measures and module 
implementation. As described in the next chapter, CRESST identified full-length modules that i3 
teachers created and/or adapted and taught in their classrooms. While all teachers may have 
taught the module(s) in their classrooms, we took the presence of uploaded student work as 
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evidence of teachers having done so. Figure 4.1 displays the mean number of CoreTools views, 
edits, and comments for the 156 participants who completed and taught at least one full-length 
LDC module and their 108 peers for whom we do not have evidence of full module 
implementation. As shown, teachers who completed and taught full-length LDC modules 
exhibited considerably more engagement with CoreTools than their peers, across all three 
metrics. The results suggest that participants who engage deeply with the module building 
platform are more likely to complete and teach LDC modules. 


Figure 4.1 


Mean Number of Coretools Views, Edits, and Comments Made by Teachers Who Did and Did Not 
Complete and Teach Modules 
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™@ Teachers who completed and taught modules (n=156) 


™ Teachers who did not complete and teach modules (n=108) 


4.4 Summary of Results 


While descriptive analyses of CoreTools user behavior showed evidence of broad 
engagement with LDC’s module building platform, the depth of that engagement varied greatly 
across users. Nearly all 2018-2019 LDC participants, including principals, created a CoreTools 
user account, and the vast majority of teachers in PLCs viewed and edited modules. The 
average teacher, however, only made 14 edits to modules within CoreTools. While this might 
be interpreted as evidence that teachers’ engagement in instructional design was somewhat 
limited, the CoreTools data do not provide any measures of instructional design work 
conducted offline by individuals or groups of teachers. In addition, our examination of editing at 
the individual level is unable to capture teachers working together on a module in one user 
account. 
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Furthermore, this descriptive analysis provides a broad rather than deep look, and does 
not take into account LDC expectations that teachers will do very little design work on their first 
module and extensive instructional design on their later modules. Teacher leaders on average 
engaged at higher levels than their peers not playing a leadership role in the LDC intervention. 
Cohort 2 teachers engaged at higher levels than their peers, while teachers newly entering the 
LDC program 2018-2019 engaged at lower levels than their peers. Participants teaching 
secondary ELA and history/social studies also engaged at higher levels than their peers in 
secondary science and at the elementary level. 


75 


5.0 Module Artifact Analysis 


This section presents results for the analysis of modules submitted during the 2018-2019 
school year. We first present overall results across the grade bands. This is followed by separate 
results for the elementary and secondary modules. Finally, we present results of an exploratory 
analysis examining results across time for teachers who submitted modules across the last 2 
school years, the qualitative results, and a summary of the findings. Tables F1 through F4 in 
Appendix F contain the generalization theory results. 


Table 5.1 shows the overall means and standard deviations by grade band for the module 
artifact analysis. When examining the ratings, the secondary modules received higher mean 
ratings than did the elementary modules. More specifically, mean ratings for the secondary 
modules ranged from 3.45 to 3.82, indicating that on average all dimensions were moderately 
present or realized. In contrast, mean ratings for the elementary modules were in the low to 
mid threes (3.00 to 3.44). The highest means for each grade band were for Dimension 1, which 
measures the effectiveness of the writing task, and Dimension 2, which measures standards 
alignment. 


Table 5.1 


Means and Standard Deviations for the Modules 


Elementary Secondary Overall 

Dimension (n = 104) (n=51) (n=155) 
1. Effective writing task 3.44 (1.00) 3.82 (0.93) 3.57 (0.99) 
2. Standards alignment 3.27 (1.16) 3.80 (0.87) 3.45 (1.10) 
3. Fidelity to LDC instruction 3.00 (1.18) 3.45 (1.08) 3.15 (1.17) 
4. Quality instructional strategies 3.18 (1.17) 3.69 (1.07) 3.35 (1.16) 
5. Coherence/clarity of module 3.21 (1.14) 3.61 (1.00) 3.34 (1.11) 
6. Overall impression 3.15 (1.07) 3.59 (0.88) 3.30 (1.03) 
Average (Dimensions 1 to 5) 3.22 (0.92) 3.68 (0.67) 3.37 (0.87) 


Furthermore, as one would hope to find, the average summary scores for the first five 
dimensions were similar to the overall impression ratings (Dimension 6) provided by the expert 
teachers. Finally, when examining the frequencies, elementary modules tended to have a lower 
proportion of modules with ratings of 4 or greater out of a possible 5 on the dimensions (36% 
to 48% depending on the dimension) than did the secondary modules (53% to 65%). Greater 
detail about the frequencies can be found in Table F5 in Appendix F. 
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5.1 Analysis of Elementary Modules 


The following section presents descriptive results for the elementary modules 
disaggregated by content area and then by cohort. Results from the generalizability theory 
modules as well as the expanded descriptive results for the elementary modules can be found 
in Appendix F. 


Descriptive Results by Content Area 


Table 5.2 presents descriptive results for the elementary modules by content area. A few 
comparative observations can be made. First, the social studies modules received the lowest 
mean ratings on all dimensions (2.48 to 3.00). Second, mean ratings for Dimensions 1 and 2 
were highest for the science modules, and the ratings for Dimensions 3 to 6 were highest for 
the ELA modules. Third, for the ELA and science modules, the average summary scores for the 
first five dimensions were similar to the overall impression ratings (Dimension 6) provided by 
the expert raters. In contrast, for the social studies modules, the average summary score was 
0.19 higher than the mean for Dimension 6. Finally, the social studies modules had the lowest 
proportion of ratings of 4 or greater out of a possible 5 on each dimension when compared to 
the other content results (see Table F6 in Appendix F). 


Table 5.2 


Means and Standard Deviations for the Elementary Modules by Content Area 


ELA Science Social studies 
Dimension (n = 38) (n = 43) (n = 23) 
1. Effective writing task 3.42 (0.79) 3.70 (0.94) 3.00 (1.28) 
2. Standards alignment 3.26 (0.79) 3.56 (1.26) 2.74 (1.32) 
3. Fidelity to LDC instruction 3.34 (0.99) 3.00 (1.25) 2.44 (1.16) 
4. Quality instructional strategies 3.50 (0.95) 3.23 (1.19) 2.56 (1.27) 
5. Coherence/clarity of module 3.58 (0.98) 3.21 (1.17) 2.61 (1.12) 
6. Overall impression 3.37 (0.91) 3.33 (0.92) 2.48 (1.31) 
Average (Dimensions 1 to 5) 3.42 (0.78) 3.34 (0.84) 2.67 (1.09) 


Descriptive Results by Cohort 


Table 5.3 presents descriptive results for the elementary modules by cohort, and there 
are three Cohort 1 groups (teachers starting in 2016-2017, 2017-2018, or 2018-2019) and two 
Cohort 2 groups (teachers starting in 2017-2018 or 2018-2019). Because of the large 
differences in sample sizes, we focus our discussion on the cohort groups with five or greater 
modules. 
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First, as can be seen, means were generally in the threes for each dimension. The 
exceptions included the pooled Cohort 1 results for Dimensions 3 to 5 (2.74 to 2.91) and the 
Cohort 1 2016-2017 results for Dimension 3 (2.88). Second, means were generally highest for 
the first two dimensions, which focused on the effective writing task and standards alignment, 
and were generally lowest for Dimension 3, which rated fidelity to LDC instruction. Third, in 
comparing the Dimension 6 ratings to the average for the first five dimensions, it is apparent 
that expert teachers’ overall impressions were fairly aligned for each of the cohort subgroups. 
Finally, with the exception of Dimension 4, which measures quality instructional strategies, 
Cohort 2 2017-2018 had the lowest proportion of ratings of 4 or greater out of a possible 5 on 
each dimension when compared to the other cohort groupings (see Table F7 in Appendix F). 


Table 5.3 


Means and Standard Deviations for the Elementary Modules by i3 Cohort Grouping 


Dimensions 2016-2017 2017-2018 2018-2019 Total 
Cohort 1 n=16 n=3 n=4 n=23 
1. Effective writing task 3.50 (1.15) -- -- 3.35 (1.07) 
2. Standards alignment 3.62 (1.20) -- -- 3.30 (1.18) 
3. Fidelity to LDC instruction 2.88 (1.31) -- -- 2.74 (1.18) 
4. Quality instructional strategies 3.06 (1.29) -- -- 2.87 (1.18) 
5. Coherence/clarity of module 3.06 (1.34) -- -- 2.91 (1.20) 
6. Overall impression 3.31 (1.08) -- -- 3.00 (1.21) 
Average (Dimensions 1 to 5) 3.22 (1.03) -- -- 3.04 (0.95) 
Cohort 2 NA n=51 n=30 n=81 
1. Effective writing task NA 3.45 (0.94) 3.50 (1.07) 3.47 (0.99) 
2. Standards alignment NA 3.12 (1.14) 3.50 (1.17) 3.26 (1.16) 
3. Fidelity to LDC instruction NA 3.10 (1.14) 3.03 (1.27) 3.07 (1.18) 
4. Quality instructional strategies NA 3.26 (1.13) 3.30 (1.24) 3.27 (1.16) 
5. Coherence/clarity of module NA 3.22 (1.06) 3.43 (1.19) 3.30 (1.11) 
6. Overall impression NA 3.14 (0.94) 3.30 (1.18) 3.20 (1.03) 
Average (Dimensions 1 to 5) NA 3.23 (0.85) 3.35 (1.02) 3.27 (0.91) 


Note. Means not presented for samples of less than five. 


5.2 Analysis of Secondary Modules 


The following section presents descriptive results for the secondary modules 


disaggregated by content area and then by cohort. Results from the generalizability theory 
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modules as well as the expanded descriptive results for the secondary modules can be found in 
Appendix F. 


Descriptive Results by Content Area 


Table 5.4 presents descriptive results for the secondary modules by content area. Because 
of the small sample size for science, we only present our observations for the ELA and social 
studies modules. First, the modules for the two content areas tended to have different 
strengths. For example, the ELA modules had moderately higher means than did the social 
studies modules for Dimensions 1 and 2, which focus on the foundational skills of the effective 
writing task and standards alignment. In contrast, the social studies modules had moderately 
higher means than did the ELA modules regarding Dimensions 5 and 6, which measure 
coherence and clarity of the module and expert rater overall impression. Second, both ELA and 
social studies modules had the lowest means for Dimension 3, which rates fidelity to LDC 
instruction (3.44, 3.50). Third, the average summary scores for the social studies modules were 
similar to the overall impression (Dimension 6) ratings provided by the expert teachers. In 
contrast, the Dimension 6 ratings were somewhat low for ELA, indicating that expert raters 
were somewhat more conservative in comparison to the summary score. Finally, when looking 
at the proportion of ratings of 4 or greater out of a possible 5, percentages ranged from 50% to 
72% for ELA and from 44% to 81% for each of the social studies dimensions (see Table F8 in 
Appendix F). 


Table 5.4 


Means and Standard Deviations for Secondary Modules by Content Area 


ELA Science Social studies 
Dimension (n = 32) (n = 3) (n = 16) 
1. Effective writing task 3.97 (0.97) — 3.50 (0.82) 
2. Standards alignment 3.84 (0.95) — 3.56 (0.63) 
3. Fidelity to LDC instruction 3.44 (1.19) _ 3.50 (0.97) 
4. Quality instructional strategies 3.72 (1.17) _ 3.75 (0.86) 
5. Coherence/clarity of module 3.56 (1.16) — 3.88 (0.50) 
6. Overall impression 3.53 (1.02) _ 3.75 (0.58) 
Average (Dimensions 1 to 5) 3.71 (0.77) = 3.64 (0.51) 


Note. Means not presented for samples of less than five. 


Results by Cohort 


Table 5.5 presents descriptive results for the secondary modules for the three i3 Cohort 1 
groupings (2016-2017, 2017-2018, 2018-2019) and two i3 Cohort 2 groupings (2017-2018, 
2018-2019). Because of the large differences in sample sizes, we focus our discussion on the 
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cohort groupings with five or greater modules. First, all of the Cohort 2 results were in the mid- 
to high threes. This was true for each subgroup as well as the pooled Cohort 2 results. In 
contrast, the Cohort 1 results had some means below 3 or greater than 4. For example, the 
pooled Cohort 1 results for standards alignment (Dimension 2) was 4.08. Second, the summary 
scores calculated across Dimensions 1 to 5 were somewhat higher than the Dimension 6 means, 
indicating that expert raters were more conservative in their impressions of overall quality. 
Finally, when looking at the proportion of ratings of 4 or greater out of a possible 5, 
percentages were generally 50% or greater on each of the dimensions for each cohort grouping 
(see Table F9 in Appendix F). 


Table 5.5 
Means and Standard Deviations for the Secondary Modules by i3 Cohort Grouping 


Dimension 2016-2017 2017-2018 2018-2019 Total 
Cohort 1 n=4 n=6 n=2 n=12 
1. Effective writing task — 3.33 (1.03) — 3.92 (1.00) 
2. Standards alignment _ 4.17 (0.75) _ 4.08 (0.90) 
3. Fidelity to LDC instruction — 2.83 (0.75) — 3.25 (1.14) 
4. Quality instructional strategies — 4.00 (0.63) _ 3.75 (1.14) 
5. Coherence/clarity of module _ 4.00 (0.63) — 3.67 (1.15) 
6. Overall impression _ 3.50 (0.55) _ 3.58 (1.08) 
Average (Dimensions 1 to 5) —_ 3.67 (0.53) — 3.73 (0.82) 
Cohort 2 NA n=26 n=13 n=39 
1. Effective writing task NA 3.96 (0.92) 3.46 (0.88) 3.80 (0.92) 
2. Standards alignment NA 3.69 (0.93) 3.77 (0.72) 3.72 (0.86) 
3. Fidelity to LDC instruction NA 3.42 (1.03) 3.69 (1.18) 3.51 (1.07) 
4. Quality instructional strategies NA 3.65 (1.06) 3.69 (1.11) 3.67 (1.06) 
5. Coherence/clarity of module NA 3.58 (1.03) 3.62 (0.87) 3.59 (0.96) 
6. Overall impression NA 3.58 (0.81) 3.62 (0.87) 3.59 (0.82) 
Average (Dimensions 1 to 5) NA 3.66 (0.63) 3.65 (0.67) 3.66 (0.63) 


Note. Means not presented for samples of less than five. 


5.3 Exploratory Analysis of Modules 


As previously noted, in order to examine potential growth among returning teachers, 
ratings were compared for teachers who were the first author and submitted complete 
modules in both the 2017-2018 and 2018-2019 school years. The following presents 
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descriptive results for the last complete module submitted by these teachers. Tables 5.6 and 


5.7 present results for the 40 elementary teachers and 27 secondary teachers who submitted 


complete modules during both school years. 


Table 5.6 


Means and Standard Deviations for the Exploratory Analysis of Modules 


Dimension 2017-2018 2018-2019 Change 
Elementary (n = 40) 
1. Effective writing task 4.25 (0.78) 3.37 (0.90) -0.88 (1.07) 
2. Standards alignment 3.95 (0.85) 3.18 (1.24) -0.78 (1.51) 
3. Fidelity to LDC instruction 3.95 (1.28) 3.15 (1.23) -0.80 (1.11) 
4. Quality instructional strategies 3.85 (1.31) 3.40 (1.10) -0.45 (1.48) 
5. Coherence/clarity of module 3.85 (1.14) 3.45 (1.04) -0.40 (1.03) 
6. Overall impression 3.83 (1.20) 3.38 (0.90) -0.45 (1.20) 
Average (Dimensions 1 to 5) 3.97 (0.93) 3.31 (0.88) -0.66 (0.94) 
Secondary (n = 27) 
1. Effective writing task 3.59 (1.22) 4.04 (0.94) 0.44 (1.45) 
2. Standards alignment 3.48 (0.89) 3.89 (0.93) 0.41 (1.37) 
3. Fidelity to LDC instruction 3.70 (1.07) 3.30 (0.95) -0.41 (1.37) 
4. Quality instructional strategies 3.85 (1.06) 3.78 (1.09) -0.07 (1.24) 
5. Coherence/clarity of module 3.81 (1.14) 3.78 (0.89) - 0.04 (1.12) 
6. Overall impression 3.81 (0.79) 3.70 (0.78) -0.11 (1.09) 
Average (Dimensions 1 to 5) 3.69 (0.73) 3.76 (0.66) 0.07 (0.88) 
Overall (n = 67) 
1. Effective writing task 3.99 (1.02) 3.64 (0.96) -0.34 (1.39) 
2. Standards alignment 3.76 (0.89) 3.46 (1.17) -0.30 (1.56) 
3. Fidelity to LDC instruction 3.85 (1.20) 3.21 (1.12) -0.64 (1.23) 
4. Quality instructional strategies 3.85 (1.21) 3.55 (1.10) -0.30 (1.39) 
5. Coherence/clarity of module 3.84 (1.14) 3.58 (0.99) -0.25 (1.08) 
6. Overall impression 3.82 (1.04) 3.51 (0.86) -0.31 (1.16) 
Average (Dimensions 1 to 5) 3.86 (0.86) 3.49 (0.82) -0.37 (0.98) 
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When examining the elementary results, means for each dimension were greater for the 
2017-2018 school year than for the 2018-2019 school year (see Table 5.6). More specifically, 
means decreased by nearly one point (-0.78 to -0.88) for Dimensions 1 to 3, which measure the 
effective writing task, standards alignment, and fidelity to LDC instruction. In contrast, means 
decreased by about one half-point for Dimensions 4 to 6, which measured quality instructional 
strategies, coherence and clarity, and overall impression. As a result, mean results went from 
representing dimensions that were generally sufficiently present or realized to moderately 
present or realized. When examining results further, 25% of teachers showed no or positive 
change on Dimension 1 and about half showed no change or positive change on the other five 
dimensions (see Table 5.7). Furthermore, the largest growth was for Dimension 4, quality 
instructional strategies), with 12 or 30% of teachers showing positive change across the 2 years. 
The lowest positive growth was for the effective writing task (Dimension 1) and fidelity to LDC 
instruction (Dimension 2), with only five or 12.5% showing increased ratings for either of these 
dimensions. 
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Table 5.7 


Counts and Percentages for the Exploratory Analysis of Modules 


Negative No change Positive 

Dimension # % # % # % 
Elementary (n = 40) 
1. Effective writing task 30 75.0 5 12.5 5 12.5 
2. Standards alignment 20 50.0 13 32.5 7 17.5 
3. Fidelity to LDC instruction 22 55.0 13 32.5 5 12.5 
4. Quality instructional strategies 22 55.0 6 15.0 12 30.0 
5. Coherence/clarity of module 20 50.0 10 25.0 10 25.0 
6. Overall impression 21 52.5 11 27.5 8 20.0 
Average (Dimensions 1 to 5) 27 67.5 5 12.5 8 20.0 
Secondary (n = 27) 
1. Effective writing task 9 33.3 4 14.8 14 51.9 
2. Standards alignment 8 29.6 6 22.2 13 48.1 
3. Fidelity to LDC instruction 10 37.0 13 48.1 4 14.8 
4. Quality instructional strategies 9 33.3 10 37.0 8 29.6 
5. Coherence/clarity of module 10 37.0 8 29.6 9 33.3 
6. Overall impression 11 40.7 8 29.6 8 29.6 
Average (Dimensions 1 to 5) 12 44.4 2 7.4 13 48.1 
Overall (n = 67) 
1. Effective writing task 28 41.8 9 13.4 19 28.4 
2. Standards alignment 28 41.8 19 28.4 20 29.9 
3. Fidelity to LDC instruction 32 47.8 26 38.8 9 13.4 
4. Quality instructional strategies 31 46.3 16 23.9 20 29.9 
5. Coherence/clarity of module 30 44.8 18 26.9 19 28.4 
6. Overall impression 32 47.8 19 28.4 16 23.9 
Average (Dimensions 1 to 5) 39 58.2 7 10.4 21 31.3 


Secondary results show a more positive picture than the one found for the elementary 
teachers (see Tables 5.6 and 5.7). First, results for most of the dimensions were almost flat or 
showed improvement. More specifically, means increased by about one half-point for 
Dimension 1 and Dimension 2, and went down by 0.11 or less for Dimensions 4 to 6. Only 
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Dimension 3, which measures fidelity to LDC, showed a moderate drop (-0.41). Second, 
individual teachers were more likely to show improvement across the 2 school years than they 
were to get lower scores (Table 5.7). For example, only about one third of teachers received a 
lower score on one or more of the first five dimensions, and about half of teachers showed 
improved ratings on Dimension 1 and/or Dimension 2. In other words, teachers showed the 
most improvement in the foundational aspects of the modules including the writing task and 
standards alignment. 


We also calculated results for the Cohort 1 and Cohort 2 teachers at the elementary and 
secondary levels. As with the overall results for each grade band, little positive growth was 
found. In addition, the means at each time point were generally higher for the Cohort 2 
elementary modules and the Cohort 1 secondary modules. Despite this, these results should be 
considered tentative because of the large differences in sample size between the cohorts. See 
Appendix F for further results. 


5.4 Qualitative Results 


Expert raters were asked to write notes about the reasoning behind their ratings for each 
module. This was done to assist in the consensus making process for all modules coded by more 
than one expert teacher, to shed light on the ratings in general, and to suggest potential ways 
to improve ratings in the future. The following presents key themes that emerged concerning 
Dimensions 1 through 5, as well as our expert raters’ overall impression. 


Dimension 1: Effective Writing Task 


While more than 80% of the modules received ratings of 3 to 5 on this dimension, the 
writing task was not always realized. Modules that received low ratings tended to have less 
information for the expert raters to evaluate. For example, many modules lacked a background 
for students, an extension activity, and/or summary information on the coversheet. When 
provided, these components were not always clear and, in some cases, did not match the rest 
of the module. 


Dimension 2: Standards Alignment 


While standards alignment was generally good, higher ratings of 4 or 5 were more 
common for the secondary modules than for the elementary ones. When providing lower 
ratings for this dimension, the expert raters noted the following issues. First, modules tended to 
have reading standards, but were less likely to have key writing, science, or social studies 
standards that matched the task and instruction. Second, some of the modules had standards 
at different grade bands—primary and upper elementary or elementary and secondary—but 
did not always provide an explanation for this misalignment (e.g., special needs classes, 
remedial coursework, etc.). Third, some modules included standards from other states. 
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Dimension 3: Fidelity to LDC Module Instruction 


One of the goals of Dimension 3 is to examine the distribution of activities and time spent 
on the four stages of the LDC instructional practice (preparation for the task, reading process, 
transition to writing, and writing process). Modules that received low ratings tended to be 
missing skills and mini-tasks that our expert teachers thought were necessary to meet the 
writing task and standards. In some cases, modules had skills listed, but did not provide a mini- 
task, or the mini-task(s) provided had scant information about the planned instruction. Low- 
rated modules also lacked additional information in the module summary, background, 
extension, and/or reflection from which to judge the reason(s) for the omissions. 


Dimension 4: Quality Instructional Strategies 


Modules that received high ratings of 4 or 5 for this dimension were noted to be thorough 
and detailed, and were sometimes noted to have student work that reflected this quality. In 
contrast, modules that received low ratings of 1 or 2 often had minimal details and/or student 
work that did not reflect the strategies. In some instances, expert teachers also noted that 
instructional strategies did not match the task, were not specific to the lesson, and/or did not 
match the grade level or grade band specified. Finally, raters noted that it was more difficult to 
determine whether the strategies were realized when only one student work sample was 
attached, or the attachments only reflected one level (e.g., only advanced). 


Dimension 5: Coherence and Clarity of Module 


Modules that received low ratings of 1 or 2 for Dimension 5 tended to have weaknesses 
across the other dimensions and components. For example, expert raters tended to notice 
issues with the quality of the writing task, standards alignment, skills, and instructional 
strategies. Raters also noted issues with student work not matching other components, or 
providing limited evidence for the goals of the lesson being met. 


Dimension 6: Overall Impressions 


Modules that received the highest rating of 5 for Dimension 6 were frequently described 
in very positive terms. Expert raters called these modules strong, focused, great, well planned, 
and thorough. In addition, during debriefings expert raters talked about how these modules 
were ready to implement. In contrast, modules that received low ratings of 1 or 2 tended to 
receive comments about being incomplete or missing many or most of the expected 
components. In some cases, comments started out stronger, but raters found that the skills, 
instruction, and/or student work did not realize the potentials of the module. 


5.5 Summary of Results 


Based on the primary analyses, the program as a whole did not show growth in the quality 
of modules between 2017-2018 and 2018-2019. More specifically, average quality for the 
elementary modules went down by nearly a point (4.11 to 3.22) and the quality of the 
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secondary modules stayed basically the same (3.68 to 3.61). In other words, elementary 
modules went from being sufficiently present or realized to being moderately present or 
realized, and secondary modules stayed moderately present or realized. Despite this, it is 
notable that more than one third of the elementary modules (36% to 48%) and more than half 
of the secondary modules (53% to 65%) received ratings of 4 or 5 on the dimensions, 
representing features that are sufficiently to fully present or realized. 


Differences in quality were also found when examining results by subgroup. For example, 
the elementary sample tended to show higher quality for ELA or science modules, and for those 
submitted by teachers at Cohort 2 schools. In contrast, differences were found by dimension 
for the secondary modules. More specifically, the highest means for Dimensions 1 and 2, which 
focus on the writing task and standards, were found for secondary ELA and Cohort 1. 
Furthermore, the highest means for Dimension 5, which measures coherence and clarity, were 
found for the secondary social studies modules, as well as those developed by Cohort 1 
teachers who began in 2017-2018. Despite this, because of the variation in sample sizes, these 
differences should be considered tentative. 


An exploratory analysis was also conducted to examine growth in performance for the 40 
elementary teachers and 27 secondary teachers who submitted complete modules with 
student work during both the 2017-2018 and 2018-2019 school years. As with the primary 
analysis, the longitudinal analysis showed different results for the two grade band samples. 
More specifically, module quality for the elementary teachers went down by nearly one point 
(0.78 to 0.88) on Dimensions 1 to 3 and nearly a half-point on Dimensions 4 to 6. In contrast, 
the secondary teachers showed a moderate drop (0.41) in quality for Dimension 3, and was flat 
for Dimensions 4 to 6. More positively, secondary teachers did show growth of nearly half a 
point for Dimensions 1 and 2, indicating that they became more successful at developing their 
writing tasks and aligning standards. 


86 


6.0 Fidelity of Implementation Analysis 


This chapter describes the results of a broad analysis of the LDC schools’ fidelity of 
implementation to the LDC model. As part of the Department of Education’s requirements, 
CRESST and LDC collaborated to create a fidelity matrix to measure schools’ fidelity of 
implementation across four key components of the LDC model. Within the matrix, each key 
component is measured via multiple indicators, with 14 indicators total across the key 
components. The key components, as depicted in the LDC logic model (see Figure 1.1) and 
described in Chapter 1 are (a) common planning time for LDC professional learning community 
with synchronous coach support; (b) asynchronous support from LDC coaches; (c) teacher 
implementation activities; and (d) leadership support. 


The fidelity matrix can be found in Appendix G. The matrix outlines a process whereby 
fidelity scores on each indicator are computed for each school and for the program as a whole. 
The matrix also produces a score for whether the program met fidelity for each of the four 
components. Although the process produces school-level fidelity scores for all indicators, the 
initial unit of implementation is either teacher, module, or school depending on the indicator. 
LDC and CRESST staff worked collaboratively to construct the fidelity matrix, with LDC staff 
setting the thresholds for adequate implementation on each indicator and at each level of 
implementation (teacher/module, school, and program). 


Data for these 14 fidelity indicators come from four sources. Three indicators are based 
on data collected via CRESST’s teacher survey. Data on these selected survey items are used to 
produce school- and program-level fidelity scores. Three indicators are based on data from the 
PLC reflection form collected by LDC, which was filled out by teacher leaders at the school level 
and used to capture information on both full PLC sessions and one-on-one planning and 
progress calls between teacher leaders and coaches. Two indicators are based on LDC 
administrative records capturing attendance by teacher leaders and administrators at in-person 
meetings occurring four times per year. Finally, six indicators are based on LDC’s CoreTools 
analytic data capturing a variety of behaviors including teachers’ viewing of LDC online course 
content in the LEARN portion of CoreTools, editing of modules, and uploading of student work; 
and coaches’ commenting and providing peer review on modules. This report explores and 
presents the overall amount of editing and commenting behavior of teacher participants in 
Chapter 4. Here we focus on whether participants edited key portions of the module, as well as 
the comments provided by coaches on modules as a measure of their asynchronous feedback. 


In this chapter, we first present the overall results for the school- and program-level 
fidelity scores for all the indicators. We then provide more detailed descriptive results with 
sections for each of the four key components and subsections for each of the 14 fidelity 
metrics. Next, we compare fidelity of implementation results to the prior year (2017-2018). 
Finally, we summarize results. 
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6.1 School- and Program-Level Fidelity of Implementation Scores 


Table 6.1 provides a summary for how LDC schools performed across the four key 
components and 14 indicators of fidelity of implementation. As described in detail in the fidelity 
matrix in Appendix G, schools were rated on 4-point scales (from 0 to 3) on each indicator. At 
the school level, adequate implementation was defined as a score of at least 2. For many of the 
indicators, the 4-point scale is based on the proportion of teachers who met a certain 
implementation threshold. In order for the program as a whole to meet fidelity on a particular 
indicator, a designated proportion of schools had to meet the school-level threshold of 2. For 
example, program-level fidelity for the Key Component 2 indicators were each met if half or 
more of the schools met fidelity. For all the other indicators (under Key Components 1, 3, and 
4) program-level fidelity was met if three quarters of schools met fidelity. 


As displayed in Figure 6.1, there was great variation across schools in their success in 
meeting the implementation thresholds. Schools met fidelity on a range of between two and 11 
indicators. On average schools met fidelity on 5.5 indicators. A total of 16 schools met fidelity 
on six or more indicators. Furthermore, no schools met fidelity on more than 11 indicators, 
which suggests that all participating schools had room for growth. 


Figure 6.1 


Frequency Distribution of Schools Reaching Adequate Implementation on Different Numbers of Fidelity 
Indicators 
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When examining the overall results concerning fidelity of implementation, a majority of 
the schools were not able to meet the adequate implementation thresholds (see Table 6.1). For 
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fidelity to be met on a key component, the fidelity matrix requires that fidelity be met on each 
of the indicators for that key component. As a result, the analysis for 2018-2019 concludes that 
fidelity was not met for the four key components across the LDC schools. 


Examining the fidelity scores by indicator, we found that adequate implementation at the 
program level was met on just two of the 14 indicators: teacher perception of the helpfulness 
of coach written feedback on modules and teacher leader attendance at quarterly in-person 
teacher leader meetings. A third indicator, which focuses on coach comments on modules, was 
on the cusp of meeting fidelity. A majority of the schools also met the threshold for perceived 
effectiveness of engagement in PLC on teacher competencies and administrator attendance at 
quarterly in-person administrator meetings, although those proportions were not sufficient for 
the program to be labeled as having adequate implementation on those indicators. For all the 
other indicators, a minority of the schools met the threshold. 
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Table 6.1 
Summary Table of School- and Program-Level Fidelity Scores by Indicator in 2018-2019 


% of schools Program 
met 
Key component Indicator With no data AtLevelO AtLevel1 AtLevel2 AtLevel3 fidelity? 

Key component 1: Common Teacher attendance at weekly PLC meetings 9 38 6 18 29 No 
planning time for LDC Amount of time spent on LDC during 0 0 56 41 3 No 
professional learning common planning time 
community with oo 

Exposure to LDC LEARN content during first 0 85 3 6 6 No 
synchronous coach support | ; 

instructional cycle 

Exposure to LDC LEARN content during 0 88 0 6 6 No 

second instructional cycle 

Perceived effectiveness of engagement in 3 18 18 27 35 No 

PLC on teacher competencies 
Key component 2: Coach comments on modules 0 38 12 12 38 No? 
Asynchronous support from Coach formative peer review on modules 94 3 0 No 
LDC coaches ; 

Teacher perception of the helpfulness of 35 20 38 Yes 

coach written feedback on modules 
Key component 3: Teacher Module editing 0 74 9 12 6 No 
implementation activities Module implementation 88 6 3 No 
Key component 4: Frequency of coach/teacher leader monthly 0 6 53 29 12 No 
leadership support at meetings 
different levels Administrator attendance at quarterly in- 0 6 24 24 47 No 

person administrator meetings 

Teacher leader attendance at quarterly in- 0 0 12 21 68 Yes 

person teacher leader meetings 

Principal mini-task observation 3 47 6 18 27 No 


*Metric was on the cusp of meeting fidelity at the program level in 2018-2019. 
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6.2 Key Component 1: Common Planning Time for LDC Professional Learning 
Community with Synchronous Coach Support 


Schools varied greatly in their level of fidelity on the five indicators of Key Component 1, 
as shown in Figure 6.2. Schools were most successful with regard to teachers’ perceptions of 
the impact of PLC engagement on their skills (perceived effectiveness of engagement in PLC on 
teacher competencies), with 21 out of 34 (62%) of the schools meeting adequate 
implementation on the indicator. A substantial minority of the schools also met fidelity on 
teacher attendance at weekly PLC meetings (47%) and amount of time spent on LDC during 
common planning time (44%). Meeting fidelity on the indicators of exposure to LDC LEARN 
content was uniformly challenging for most schools, with only a handful of schools reaching the 
adequate implementation threshold in each case. 


Teacher Attendance at Weekly PLC Meetings 


PLC reflection data show that schools varied greatly in terms of the number of times their 
PLCs met over the course of the 2018-2019 school year. The number of sessions ranged from 
four to 20 times, with PLCs on average meeting 12.6 times. This is important context for the 
interpretation of the teacher attendance at weekly PLC meetings fidelity indicator. The indicator 
measures the proportion of teachers who regularly attended sessions, but does not incorporate 
the number of times that the PLC met across the year. Success in these two facets are 
somewhat correlated, but there were many cases where teacher attendance rates were high 
but the school met a below average number of times. Likewise, there were cases where schools 
met a large number of times but individual attendance rates were poor. 


Attendance rates across the full population of PLC participants varied a great deal, ranging 
from zero to 100% and averaging 78%. Classroom teachers playing an LDC teacher leader role 
(n = 33, 91%) attended at higher levels than did classroom teachers not playing a leadership 
role (n = 218, 76%). Teacher leaders in out-of-classroom assignments (n = 5) attended least 
frequently (64%). The adequate implementation threshold for individual teacher attendance 
was 80% or greater, and 168 out of 256 PLC participants (66%) reached this threshold. As seen 
in Figure 6.2, while a majority of teachers in a majority of schools reached this threshold, there 
was a very substantial minority of schools that experienced low attendance rates. More 
specifically, in nearly 40% of schools, less than a quarter of the teachers reached the adequate 
implementation threshold of an 80% attendance rate. 
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Figure 6.2 
Number of Schools Reaching Different Fidelity Levels for Each Key Component 1 Indicator 
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Amount of Time Spent on LDC During Common Planning Time 


This metric is based on PLC reflection data for the subset of PLC sessions that coaches 
joined either digitally or in person. Data on sessions and their lengths were missing for three of 
the participating schools. Teacher leaders in the remaining 31 schools reported that 33 sessions 
were under 45 minutes long (9%), 167 sessions were between 45 and 59 minutes (44%), 151 
sessions were between 60 and 74 minutes (39%), and 32 sessions were between 75 and 90 
minutes (8%). 


School-level scores were based on the modal (most common) response on the PLC 
reflection form. For 19 of the schools that response was 45-59 minutes, for 14 schools the most 
common response was 60-74 minutes, and one school met most frequently for 75-90 minutes. 
Because of the nature of the data, we cannot ascertain the exact length of the sessions. As such 
it is possible that many of the sessions recorded as 45-59 minutes came quite close to the 
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desired 60 minutes of PLC time. Nevertheless, according to the thresholds set, a majority of the 
schools did not meet fidelity on this indicator. 


Exposure to LDC LEARN Content During First and Second Instructional Cycles 


Here we report on the two indicators measuring PLC participants’ engagement with the 
LEARN online course content (instructional courses that help LDC participants learn how to 
navigate an LDC instructional cycle). Nearly all (94%) teachers and teacher leaders participating 
in LDC PLCs in 2018-2019 viewed at least one session in each of two instructional cycles. 
Overall the full population of teachers and teacher leaders on average were exposed to 41% of 
the LEARN sessions in the first instructional cycle and 33% of the LEARN sessions in the second 
instructional cycle. This represents a marked improvement over the prior year, when 
participants on average were exposed to 18% of LEARN sessions in the first instructional cycle 
and 9% in the second instructional cycle. Despite this improvement in the average exposure, 
the great majority of teachers and teacher leaders were still not meeting the fidelity threshold 
of exposure to 60% of instructional cycle sessions. Only 27% of participants met this threshold 
for the first instructional cycle, and 18% of participants met it for the second instructional cycle. 


Overall a total of four schools met the adequate implementation threshold for the first 
instructional cycle (scoring at Fidelity Levels 2 and 3) and four schools met the threshold for the 
second instructional cycle (three of the four schools were the same across both cycles). Nearly 
all the remaining schools scored at the lowest fidelity score level. 


Perceived Effectiveness of Engagement in PLC on Teacher Competencies 


This indicator is based on the 224 teachers who provided data on Question 30 in the 
teacher survey (see Appendix A), which asks teachers to report how much their skills had 
improved in a number of areas aligned with LDC core competencies. Here we base fidelity 
levels on an index averaging the seven survey items (see Question 30 in Appendix A), which are 
each measured on a scale of 1 to 4. The threshold for adequate implementation at the teacher 
level is an index score of 3, corresponding to the survey response reporting moderate 
improvement in the skill area. Across the whole sample, six teachers (3%) had scores of 1 to less 
than 2 (no to a little improvement), 47 (21%) had scores of 2 to less than 3 (a little to moderate 
improvement), 121 (54%) had scores of 3 to less than 4 (moderate to great improvement), and 
50 (22%) had scores of 4 (great deal of improvement). Over three quarters of respondents in 
the sample reached the adequate implementation threshold score of 3 or greater on the index. 


There was a great deal of variation in survey responses both within and across schools. In 
12 schools, half or fewer of the teachers reported a moderate effect on the competencies. On 
the other hand, 100% of respondents in 10 schools reported a moderate or greater effect. The 
variation can be clearly seen in Figure 6.2. Despite the large proportion of teachers who 
reported that LDC had an impact on their competencies, less than two thirds of the schools had 
three quarters or more of their teachers report a moderate impact. 
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6.3 Key Component 2: Asynchronous Support From LDC Coaches 


As with Key Component 1, there was a great deal of variation across schools in their level 
of fidelity on the three Key Component 2 indicators, as shown in Figure 6.3. Data indicated high 
fidelity on the teacher perception of the helpfulness of coach written feedback on modules 
indicator, one of only two indicators for which the program as a whole met fidelity. Twenty out 
of 34 schools (59%) met fidelity on this indicator. In addition, the program nearly met fidelity on 
coach comments on modules, with half of schools meeting the adequate implementation 
threshold. Coach peer review was not a broadly used tool, and as a result only one school met 
fidelity on this indicator. 


Figure 6.3 
Number of Schools Reaching Different Fidelity Levels for Each Key Component 2 Indicator 
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Coach Comments on Modules 


Results on this indicator are based on the universe of modules edited at least once during 
the 2018-2019 school year and linked to a LEARN instructional cycle. In total there were 158 
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modules linked to courses across 33 of the 34 schools (one school did not have any teachers 
edit modules at least once and link to a LEARN instructional cycle), with the number of linked 
modules per school ranging from one to 19. Those modules in turn received between zero and 
14 comments from coaches. An adequate level of feedback on a linked module was defined as 
having received two or more coach comments, and 94 out of the total 158 modules (60%) met 
that threshold. The proportion of modules within schools that met the two-comment threshold 
varied a great deal across schools, which accounts for the variation in school-level fidelity 
scores as shown in Figure 6.3. 


Coach Formative Peer Review on Modules 


Results on this indicator are based on the universe of teachers and teacher leaders who 
coauthored at least one linked course. That population includes 182 participants across 33 of 
the 34 schools (one school did not have any teachers with modules linked to courses). At the 
teacher level, fidelity was defined as having received at least one peer review from a coach. 
Only 39 teachers received at least one peer review from their coach, with only one of the 
schools meeting the school-level fidelity threshold. 


Teacher Perception of Helpfulness of Coach Written Feedback on Modules 


Across 224 teacher survey respondents, 154 teachers (69%) reported using written 
feedback on modules and finding it moderately to very helpful. Across schools, the proportion 
of teachers who reported at least moderate usefulness ranged from zero to 100% with a mean 
of 71%. Twenty out of 34 schools met the adequate implementation threshold, and as a result 
this was one of two indicators for which the program as a whole met the fidelity requirement. 


6.4 Key Component 3: Teacher Implementation Activities 


Here we report on indicators related to teachers’ engagement with the process of 
designing and implementing LDC modules. As shown in Figure 6.4, schools were more 
successful in terms of meeting thresholds for the proportion of their teachers who engaged 
deeply in the design process by editing key portions of modules, with almost a fifth of schools 
meeting fidelity on this indicator. In contrast, only three of the schools had three quarters or 
more of their teachers implement two modules, as measured by the uploading of student work 
samples. 


Module Editing 


As outlined in the fidelity matrix, teacher fidelity levels are built in a stepladder fashion. 
Teachers meeting Fidelity Level 1 were required to edit at least one task in a module. Teachers 
meeting Fidelity Level 2 (adequate implementation) were required to meet Fidelity Level 1 and 
edit either standards or texts in at least one module; teachers at this level did not edit either 
skills/mini-tasks or rubrics. Finally, teachers meeting Fidelity Level 3 met the previous two 
requirements (editing at least one task and standards or texts) and also edited skills/mini-tasks 
and/or the rubric. 
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A total of 138 participants (51%) failed to meet the threshold of editing the teaching task 
in one module and therefore implemented at a fidelity level of zero. Twenty participants (7%) 
edited a task, but did not edit standards or texts, and therefore were at Fidelity Level 1. A total 
of 111 participants (41%) met the adequate implementation threshold, with 25 (9%) scoring at 
Level 2 (edited standards or texts) and 86 (32%) scoring at Level 3 (edited standards or texts 
and skills/mini-tasks or rubric). As a result, overall less than half of participants met the teacher- 
level adequate implementation threshold, and less than one out of five schools met the school- 
level threshold for adequate implementation. It is important to note that in response to 
teachers collaborating on and implementing common modules in schools, LDC refined its data 
collection in the summer of 2018 to include tracking engagement data on modules that 
teachers collaborated on. The fidelity matrix and CRESST’s analyses of editing data were not 
designed to capture this shift toward a collaborative model of instructional design in CoreTools, 
and therefore may not fully capture engagement in the design process. 


Figure 6.4 
Number of Schools Reaching Different Fidelity Levels for Each Key Component 3 Indicator 
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Module Implementation 


For this indicator, our sample includes the 264 classroom teachers who participated in 
LDC in 2018-2019 (as the metric involves classroom implementation, out-of-classroom staff are 
excluded). In this case, the uploading of student work served as a proxy for whether the teacher 
implemented a module in their classroom. The number of modules with uploaded student work 
ranged from 0 to 4 with a mean of just over 1. One hundred eight teachers (40%) didn’t upload 
student work to any modules. Seventy (27%) uploaded student work to one module, and 63 
(24%) uploaded to two modules. Thirty-three teachers (9%) uploaded student work to three or 
more modules. Overall, just one third of teachers met the adequate implementation threshold, 
and as a result only 9% of schools met fidelity on the indicator. 


6.5 Key Component 4: Leadership Support at Different Levels 


As depicted in Figure 6.5, there was a high level of fidelity with regard to the attendance 
of teacher leaders at quarterly in-person meetings. Results, however, were considerably less 
positive for the other three indicators of leadership support, with an insufficient number of 
schools meeting the adequate implementation threshold on each to meet program fidelity. 
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Figure 6.5 
Number of Schools Reaching Different Fidelity Levels for Each Key Component 3 Indicator 
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Frequency of Coach/Teacher Leader Monthly Meetings 


Teacher leaders reported meeting with their coach between zero and 19 times with a 
mean of 8.7 times, according to the PLC reflection data. The adequate implementation 
threshold was set at nine or more coach/teacher leader meetings across the school year, and 
only 14 of 34 schools (41%) met that threshold. Eighteen of 34 schools (53%) met between four 
and eight times (low implementation) and two (6%) met less than four times (very low 
implementation). 


Administrator Attendance at Quarterly In-Person Administrator Meetings 


Administrators had the opportunity to attend three administrator training events. 
Administrators in 16 schools (47%) attended all three events, in eight schools (24%) attended 
two events, in eight schools (24%) attended one event, and in two schools (6%) administrators 
attended zero events. With the adequate implementation threshold set at two of three events, 
24 out of 34 schools (71%) met the threshold. 
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Teacher Leader Attendance at Quarterly In-Person Teacher Leader Meetings 


Teacher leaders also had the opportunity to attend three teacher leader training events. 
Teacher leaders attended all three events in 23 schools (68%), attended two events in seven 
schools (21%), and attended one event in four schools (12%). With adequate implementation 
again set at two of three events, 30 out of 34 schools (88%) met the threshold. 


Principal Mini-Task Observation 


Teachers were asked to report how many times their administrator observed their 
teaching of an LDC mini-task. Thirty-eight percent of survey respondents reported never having 
been observed by an administrator when teaching an LDC mini-task, 22% reported being 
observed once, 25% reported being observed twice, and 15% reported being observed three or 
more times. Under the fidelity matrix, 75% of teachers had to be observed for the school to 
meet fidelity, and only 44% of schools met this threshold. 


6.6 Change in Fidelity of Implementation From 2017-2018 to 2018-2019 


Here we explore change over time in how the program performed on the 14 different 
fidelity metrics. Table 6.2 displays the proportion of schools that met fidelity thresholds in 
2017-2018 and 2018-2019, and whether the program as a whole met fidelity on the metric in 
each year. It is important to note that the 2 years of analysis are based on two different 
samples of schools. Eleven schools that participated in LDC in 2017-2018 did not continue with 
the program in 2018-2019. Therefore, the 2018-2019 sample includes 34 schools, compared to 
45 schools in 2017-2018. 


In each of the 2 years, the program met fidelity on two of the four metrics. Fidelity was 
met both years on teacher leader attendance at quarterly in-person teacher leader meetings, 
while the program met fidelity on administrator attendance at quarterly in-person 
administrator meetings in 2017-2018 only, and on teacher perception of the helpfulness of 
coach written feedback in 2018-2019 only. 


Overall, the percentage of schools that met fidelity increased for eight metrics and 
decreased for six metrics. Some of the most notable improvements over time were seen on the 
three survey-based fidelity metrics, with schools more likely to meet fidelity on the perceived 
effectiveness of PLCs on teacher competencies, the perceived helpfulness of written feedback 
by coaches, and the extent of principal observations. There was also evidence that the program 
was successful in exposing more teachers to instructional course content, although only a small 
proportion of schools met adequate implementation on these metrics. 
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Table 6.2 


Summary Table Displaying Fidelity of Implementation by Metric and Year 


2017-2018 


2018-2019 


% of schools 
that met fidelity Program 


% of schools 
that met fidelity Program 


threshold met threshold met 
Key component Indicator (n = 45) fidelity? (n = 34) fidelity? 

Key component 1: Teacher attendance at weekly PLC meetings 53 No 47 No 
Common planning time Amount of time spent on LDC during common planning time 38 No 44 No 
for LDC professional A dens ; 
| ; : R Exposure to LDC LEARN content during first instructional cycle 0 No 12 No 
earning community with 
synchronous coach Exposure to LDC LEARN content during second instructional 0 No 12 No 
support cycle 

Perceived effectiveness of engagement in PLC on teacher 44 No 62 No 

competencies 
Key component 2: Coach comments on modules 44 No 50 No? 
Asynchronous support Coach formative peer review on modules 9 No 3 No 
from LDC coaches : . 

Teacher perception of the helpfulness of coach written 31 No 59 Yes 

feedback on modules 
Key component 3: Module editing 27 No 18 No 
Teacher implementation \yodule implementation 13 No 9 No 
activities 
Key component 4: Frequency of coach/teacher leader monthly meetings 29 No 41 No 
Leadership support at Administrator attendance at quarterly in-person administrator 82 Yes 71 No 
different levels meetings 

Teacher leader attendance at quarterly in-person teacher 96 Yes 88 Yes 

leader meetings 

Principal mini-task observation 22 No 44 No 


*This metric was on the cusp of meeting fidelity at the program level in 2018-2019. 


100 


6.7 Exploratory Analysis of the Relationship Between Implementation Metrics 
and Student Outcomes 


Here we explore the relationship between the fidelity of implementation indicators and 
the student ELA outcome scores at both the school and teacher level in 2018-2019. In order to 
examine these relationships, we produced school and teacher residuals, which are estimates of 
individual schools’ and teachers’ contributions to student ELA scores (i.e., schools’ and 
teachers’ effectiveness in impacting student achievement), holding the other variables in the 
analysis model constant. 


Two MMMC models were used: one to produce individual estimates for each LDC school 
and one to produce individual estimates for each LDC teacher. The models utilized the analytic 
sample of Cohort 1 and Cohort 2 elementary and middle school teachers and students reported 
on in Chapter 7 (the largest sample for our primary quasi-experimental analyses). The models 
we used to produce the school and teacher residuals were identical to our outcome models 
with two exceptions: (a) the LDC treatment variable was removed from the equation; and 
(b) one classification level was removed for each of the two models. In the model to produce 
school-level estimates we removed the teacher classification level. Similarly, in the model to 
produce teacher-level estimates, we removed the school classification level. 


The school and teacher residuals from the respective models were used as the estimates 
of effectiveness, after adjustment by all the model covariates. These residuals are sometimes 
referred to as best linear unbiased predictors (BLUPs) in the relevant literature (Robinson, 
1991). Because the residuals are obtained from our analytic samples, which exclude many 
participating schools and teachers for a variety of reasons (e.g., lack of prior or outcome scores; 
teaching a subject other than ELA, social studies, or science), the resulting estimates represent 
a subset of all schools and teachers in 2018-2019. Residuals were produced for 25 out of 34 
schools (74%), and 75 out of 264 teachers (28%). 


For the school-level examination of the relationship between fidelity and outcomes, we 
included school-level versions of the 14 fidelity matrix indicators, each measured using a 4- 
point scale of 0 to 3. We examined the distribution of school-level residuals and divided them 
into three categories: nine low-achieving schools with residuals of less than -0.10, six middle- 
achieving schools with residuals between -0.10 and 0.10, and 10 high-achieving schools with 
residuals greater than 0.10. We then produced mean implementation scores on each indicator 
for the three achievement groups, which are displayed in Table 6.3. A descriptive examination 
of the school means did not reveal any consistent pattern between the implementation scores 
and achievement levels. We also conducted a one-way ANOVA to test whether the differences 
in means between the three groups were significant. These tests only found a statistically 
significant difference on one indicator, and this result may be an artifact of conducting multiple 
comparisons. Finally, the analysis provided no evidence of a link between school-level fidelity of 
implementation on the matrix indicators and school-level effectiveness. 
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Table 6.3 


Mean School-Level Fidelity Scores by Indicator and Achievement Group 


Low-achieving 


Middle-achieving 


High-achieving 


Difference in 


schools (n = 9) schools (n = 6) schools (n = 10) means 
significant at 
Key component Fidelity indicator M n M n M n 5% level 

Key component 1: Common Teacher attendance at weekly PLC meetings 1.33 9 1.17 6 1.00 8 No 
planning time for che Amount of time spent on LDC during 1.22 9 1.33 1.50 10 No 
professional learning common planning time 
community with tnd 

Exposure to LDC LEARN content during first 0.00 9 0.50 6 0.30 10 No 
synchronous coach support | ; 

instructional cycle 

Exposure to LDC LEARN content during 0.00 9 0.83 6 0.00 10 Yes 

second instructional cycle 

Perceived effectiveness of engagement in 1.78 9 2.00 6 1.50 10 No 

PLC on teacher competencies 
Key component 2: Coach comments on modules 1.78 1.50 1.40 10 No 
Asynchronous support from Coach formative peer review on modules 0.00 0.33 0.10 10 No 
LDC coaches ; 

Teacher perception of the helpfulness of 1.78 1.50 1.30 10 No 

coach written feedback on modules 
Key component 3: Teacher Module editing 1.00 0.83 0.70 10 No 
implementation activities — )oqule implementation 0.22 0.17 0.20 10 No 
Key component 4: Frequency of coach/teacher leader monthly 1.67 9 1.50 6 1.30 10 No 
Leadership support at meetings 
different levels Administrator attendance at quarterly in- 2.22 9 2.17 6 2.00 10 No 

person administrator meetings 

Teacher leader attendance at quarterly in- 2.89 9 2.50 6 2.30 10 No 

person teacher leader meetings 

Principal mini-task observation 1.33 9 1.33 6 1.00 10 No 
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Table 6.4 


Mean Values on Teacher-Level Variables Associated With Fidelity Indicators by Achievement Group 


Low-achieving 


Middle-achieving 


High-achieving Difference in 


teachers (n = 22) teachers (n = 27) teachers (n = 26) means 
significant at 
Fidelity indicator Teacher-level variable M n M n M n 5% level 
Teacher attendance at weekly PLC attendance rate 0.87 21 0.71 25 0.81 23 No 
PLC meetings 
Exposure to LDC LEARN content Exposure rate for instructional course 1 0.42 22 0.34 27 0.45 26 No 
during first instructional cycle sessions 
Exposure to LDC LEARN content Exposure rate for instructional course 2 0.30 22 0.26 27 0.37 26 No 
during second instructional cycle sessions 
Perceived effectiveness of Survey-based index measuring perceived 3.42 20 3.10 24 3.13 23 No 
engagement in PLC onteacher impact of PLC on teacher competencies 
competencies 
Coach comments on modules Number of teacher’s modules that met 0.63 11 1.00 13 0.94 16 No 
fidelity for coach commenting 
Coach formative peer review on Number of teacher’s modules that 0.32 22 0.26 27 0.12 26 No 
modules received a formative peer review 
Teacher perception of the Survey-based perception of the 3.00 20 2.58 24 2.7 23 No 
helpfulness of coach written helpfulness of coach written feedback on 
feedback on modules modules 
Module editing Module editing teacher-level fidelity score 1.32 22 1.19 27 1.23 26 Yes 
Module implementation Number of teacher’s modules with 1.32 22 1.19 27 0.77 26 No 
uploaded student work 
Principal mini-task observation Numerical version of scale asking survey 1.45 20 1.50 24 1.43 23 No 


respondents about the number of times 
their principal observed their LDC 
instruction 
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For the teacher-level analysis, we included 10 teacher variable versions of the fidelity 
matrix indicators (four of the 14 indicators are only measured at the school level and therefore 
cannot be included in the teacher-level analysis). As with the school-level analysis, we 
examined the distribution of residuals and divided them into three categories: 22 low-achieving 
schools with residuals of less than -0.05, 27 middle-achieving schools with residuals 
between -0.05 and 0.10, and 26 high-achieving schools with residuals greater than 0.05. We 
then produced mean implementation scores on each indicator for the three achievement 
groups, which are displayed in Table 6.4. 


Again, as with the school-level analysis, a descriptive examination of the means did not 
reveal any consistent pattern between implementation scores and achievement levels. We also 
conducted a one-way ANOVA to test whether the differences in means between the three 
groups were significant. These tests only found a statistically significant difference on one 
indicator, and this result may be an artifact of conducting multiple comparisons. In addition, the 
analysis provided no evidence of a link between teacher-level fidelity of implementation on the 
matrix indicators and teacher-level effectiveness. 


6.8 Summary of Results 


In summary, our analysis revealed that none of the schools met fidelity requirements on 
the four key components in 2018-2019 in the West Coast district. The ability of PLCs to set 
aside common planning time that worked for all teachers varied across schools. The frequency 
of meetings ranged broadly, as did the attendance rates of the participants. While many 
schools were able to maintain high attendance rates, a majority did not meet the desired 
attendance threshold. When PLCs did meet, they most often met for 45 to 59 minutes, 
although many sessions also lasted the desired 60 minutes or more. LDC intended for teachers 
to be exposed to key online course content within PLC time, and this year’s fidelity analysis 
showed substantial improvement at the teacher level on these metrics. The vast majority of 
schools, however, still did not meet the adequate implementation thresholds for the 
instructional course exposure metrics. A majority of schools met fidelity in terms of teachers’ 
perception of the impact of LDC on their skills, representing a marked improvement over the 
prior year; nevertheless, the program as a whole did not meet fidelity on this metric. 


The results reveal some improvement from 2017-2018 to 2018-2019 with regard to 
asynchronous support from coaches. Schools were somewhat more likely to meet fidelity 
thresholds regarding coaches’ commenting on modules, and teachers were considerably more 
likely to find written feedback helpful. Peer review remained a little used function in CoreTools. 


Results on module editing suggest that many teachers were not heavily engaged in the 
design process, with half of teachers not having edited any teaching tasks. Those that did edit 
the teaching task tended to also edit either standards or texts, and a third of teachers also 
reached the stage of editing skills/mini-tasks and the rubric. There was evidence that 60% of 
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teacher participants implemented modules in the classroom, but only about a third of teachers 
met the adequate implementation threshold of uploading student work for two modules. 


Teacher leaders continued to attend professional development meetings at high levels, 
but there was some dropoff in engagement by administrators in professional development 
meetings from 2017-2018 to 2018-2019. As a result, the program no longer met fidelity on the 
latter indicator in 2018-2019. The program was somewhat more successful in 2018-2019 in 
ensuring regular contact between teacher leaders and coaches and broad engagement by 
administrators in observing LDC classroom implementation, but overall fidelity was still not met 
on these indicators. 


Variation across schools was substantial. Some schools met fidelity on eight to 11 
indicators, and on the other end, some schools struggled with nearly every aspect of 
implementation. No strong patterns emerged for the cohort and school-level subgroups with 
Cohort 1 and Cohort 2 schools, as well as the elementary and secondary schools meeting similar 
levels of fidelity, despite great variation within each of the subgroups. 


Despite substantial variation across schools and teachers in fidelity of implementation 
across the different metrics, an exploratory analysis did not find any relationship at either the 
school or teacher level between fidelity and effectiveness in impacting student ELA scores. 
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7.0 Student Outcome Analysis 


This chapter presents the student outcome analysis we conducted to evaluate the impact 
of LDC on student learning. As described in Chapter 2, our LDC teacher samples included both 
elementary and middle school teachers in two cohorts of schools in the study district. This 
chapter explores impacts of teachers after participating in LDC for 2 years and 3 years. Our 
primary analyses of the impact of LDC after 2 years pool teachers and students from both 
cohorts of schools. These 2-year impact analyses are conducted for all elementary school 
teachers, all middle school teachers, and all teachers combined across school levels. Each 
separate analysis included matching and regression stages. Specifically, we conducted outcome 
analyses for the following groups of schools, teachers, and their students: 


1. Cohorts 1 and 2 elementary school sample: Teachers implementing LDC for 2 
consecutive years (Cohort 1: 2016-2017 and 2017-2018; Cohort 2: 2017-2018 and 
2018-2019) 


2. Cohorts 1 and 2 middle school sample: Teachers implementing LDC for 2 consecutive 
years (Cohort 1: 2016-2017 and 2017-2018; Cohort 2: 2017-2018 and 2018-2019) 


3. Cohorts 1 and 2 elementary and middle school sample: Teachers implementing LDC 
for 2 consecutive years (Cohort 1: 2016-2017 and 2017-2018; Cohort 2: 2017-2018 
and 2018-2019) 


4. Cohort 1 middle school sample: Teachers implementing LDC for 3 consecutive years 
(2016-2017, 2017-2018, and 2018-2019) 


5. Cohort 2 elementary school sample: Teachers implementing LDC for 2 consecutive 
years (2017-2018 and 2018-2019) 


6. Cohort 2 middle school sample: Teachers implementing LDC for 2 consecutive years 
(2017-2018 and 2018-2019) 


7. Additional analyses based on the Cohorts 1 and 2 middle school sample: LDC students 
with or without prior exposure to LDC teachers and LDC students with high or low LDC 
dosage in the outcome year 


LDC’s pre-grant theory of action predicted that it would take 2 years of experience with 
LDC before teachers have a measurable impact on student learning’. As such, the first four 
analyses listed above constitute this study’s primary tests of the impact of LDC. Analyses 
numbered 5 through 7 above are considered supplementary. The fifth and sixth analyses 
examine impact after 2 years for just the Cohort 2 schools, which joined a year later than the 
Cohort 1 schools, benefited from early lessons learned by the LDC team, and experienced a 
lower level of teacher attrition than did the Cohort 1 schools. The seventh analysis breaks the 
Cohorts 1 and 2 middle school sample (listed as number 2 above) into subgroups based on 


> Of note is the fact that our prior report found a positive statistically significant impact of LDC in Cohort 2 middle 
schools after one year (Wang, et al. 2019) 
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whether students were exposed to LDC in the prior year and how much LDC instruction 
students were exposed to in the outcome year. We begin this chapter by describing the process 
we used to define the LDC student samples for each analysis and to construct matched 
comparison samples. We then present descriptive statistics for the treatment and comparison 
groups. Next, we report the estimated impact of LDC on students as measured by the Smarter 
Balanced scores in ELA for our primary and supplementary analyses. Finally, we summarize and 
contextualize the results. 


7.1 LDC Sample and the Matching Process 


For each analysis, the eligible LDC sample includes all students (a) who were enrolled in 
one school campus for the entire outcome year under the instruction of at least one of the 
participating LDC teachers and (b) for whom baseline achievement scores, outcome year 
achievement scores, and demographic data were available. Achievement and demographic 
data were used in the matching process. Six sets of matching were conducted, four for the 
primary analysis samples (elementary teachers with 2 years of LDC, middle school teachers with 
2 years of LDC, combined elementary and middle school teachers with 2 years of LDC, and 
Cohort 1 middle school teachers with 3 years of LDC) and two for the supplementary analysis 
samples (Cohort 2 elementary and middle school teachers with 2 years of LDC, respectively). 
We present the matching results for the primary analysis samples here, with matching results 
for supplementary analyses displayed in Tables 11 and I2 in Appendix I. 


Cohorts 1 and 2 Elementary School Sample (Teachers With 2 Years of Implementation) 


Prior to the CEM process this LDC sample included 17 schools, 34 teachers, and 757 
students. After the student-level matching, our final pooled elementary LDC sample included 
685 students and the same number of teachers and schools prior to matching (see Table 7.1). 


Prior to matching, the potential elementary school comparison sample consisted of 566 
schools, 2,600 teachers, and 65,503 students. This comparison sample was substantially 
reduced during the first stage of matching, which identified up to five schools that closely 
resembled each of the LDC schools. After student-level CEM, a workable analytic comparison 
sample consisted of 80 schools, 230 teachers, and 685 students. 
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Table 7.1 


Before and After Matching Sample Sizes: Cohorts 1 and 2 Elementary School Analysis for Teachers With 2 
Years of LDC Implementation 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 17 35 757 566 2,600 65,503 
After school matching 17 35 757 82 333 5,986 
Stage 2 
After student matching 17 34 685 80 230 685 


Cohorts 1 and 2 Middle School Sample (Teachers With 2 Years of Implementation) 


As reported in Table 7.2, the resulting LDC sample for the pooled middle school teachers 
with 2 years of implementation included 11 schools, 54 teachers, and 3,575 students prior to 
the CEM process. After the CEM student-level matching, our final pooled middle school LDC 
sample was reduced to 3,294 students. 


Prior to matching, the potential comparison sample consisted of 117 schools, 3,330 
teachers, and 100,604 students. This comparison sample was substantially reduced during the 
first stage of matching, which identified up to five schools that most closely resembled each of 
the LDC schools. After student-level CEM, a workable analytic comparison sample of 38 schools, 
787 teachers, and 3,294 students was constructed. 


Table 7.2 


Before and After Matching Sample Sizes: Cohorts 1 and 2 Middle School Analysis for Teachers With 2 
Years of LDC Implementation 


LDC sample Comparison sample 
Stage School Teacher Student School Teacher Student 
Stage 1 
Before school matching 11 54 3,575 117 3,330 100,604 
After school matching 11 54 3,575 40 1,146 31,327 
Stage 2 
After student matching 11 54 3,294 38 787 3,294 
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Cohorts 1 and 2 Elementary and Middle School Sample (Teachers With 2 Years of 
Implementation) 


We also conducted an analysis which pooled teachers with 2 years of LDC experience 
across cohorts and school levels. As shown in Table 7.3, the resulting LDC sample included 26 
schools, 89 teachers, and 4,331 students prior to the CEM process. It should be noted that two 
of the LDC schools spanned across both elementary and secondary grades. After the CEM 
student-level matching, our final secondary LDC sample was reduced to 3,979 students. 


Prior to matching, the potential comparison sample consisted of 660 schools, 5,885 
teachers, and 147,332 students. This comparison sample was substantially reduced during the 
first stage of matching, which identified up to five schools that closely resembled each of the 
LDC schools. After student-level CEM, a workable analytic comparison sample of 111 schools, 
1,015 teachers, and 3,979 students was constructed. 


Table 7.3 


Before and After Matching Sample Sizes: Cohorts 1 and 2 Elementary and Middle School Analysis for 
Teachers With 2 Years of LDC Implementation 


LDC sample Comparison sample 
Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 26 89 4,331 660 5,885 147,332 
After school matching 26 89 4,331 115 1,476 37,253 
Stage 2 
After student matching 26 89 3,979 111 1,015 3,979 


Cohort 1 Middle School Sample for Teachers With 3 Years of LDC Implementation 


As shown in Table 7.4, the resulting Cohort 1 LDC middle school sample included three 
schools, eight teachers, and 831 students prior to the CEM process. After the CEM student-level 
matching, our final secondary LDC sample was reduced to 801 students. 


Prior to matching, the potential comparison sample consisted of 115 schools, 2,663 
teachers, and 74,727 students. This comparison sample was substantially reduced during the 
first stage of matching, which identified up to five schools that closely resembled each of the 
LDC schools. After student-level CEM, a workable analytic comparison sample of 14 schools, 
247 teachers, and 801 students was constructed. 
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Table 7.4 


Before and After Matching Sample Sizes: Cohort 1 Middle School Analysis for Teachers With 3 Years of 
LDC Implementation 


LDC sample Comparison sample 
Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 3 8 831 115 2,663 74,727 
After school matching 3 8 831 14 394 10,390 
Stage 2 
After student matching 3 8 801 14 247 801 


7.2 Primary Outcome Analysis: Descriptive Results on the Matched Analytic 
Samples 


Tables 7.5 through 7.8 present the characteristics of the LDC students and comparison 
students in the four sets of primary analyses respectively (descriptive results for matched 
samples for the supplementary analyses can be found in Tables 13 through I6 in Appendix I). 
Treatment and comparison samples matched very closely. Exact matching was achieved on 
some variables, and all demographic variables were within five percentage points. We used the 
spring 2019 Smarter Balanced ELA tests as our outcome measures in all analyses except when 
we combined Cohorts 1 and 2 schools; for those pooled analyses, spring 2018 test scores were 
used as the outcome measures for Cohort 1 students. Additionally, the Cohort 1 analyses used 
the spring 2016 Smarter Balanced ELA tests as one of the matching variables while the Cohort 2 
analyses used the spring 2017 Smarter Balanced ELA tests. 


For the baseline and outcome achievement variable, we standardized Smarter Balanced 
scale scores at each grade level relative to district performance, based on the district mean and 
standard deviation for the ELA test at each grade level. Standardizing scores in this way enables 
us to easily compare our sample’s performance relative to the district’s and to compare scores 
across grades and years more easily. A standardized scale score of zero, for example, indicates 
that the student scored at the mean for all other students in the district who took the same 
test. A standardized scale score of 1.0 meant that the student scored one standard deviation 
higher than the district mean. Conversely, a standardized scale score of -1.0 indicated that the 
student scored one standard deviation lower than the district mean. 


The pooled LDC elementary school student sample after matching was composed mainly 
of Hispanic students eligible for free or reduced-price lunches (see Table 7.5). English language 
learners represented 15% of this sample, while special education students and those classified 
as gifted represented 6.3% and 12.7% respectively. The sample was also mostly comprised of 
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students in third grade (86.0%) in the baseline year (and therefore in fifth grade in the outcome 
year), with the remainder being students in a" grade at baseline and 6" grade in the outcome 

year (some elementary schools in the study had a K—-6 grade range). Mean performance on the 
baseline year assessments was about one eighth of a standard deviation above the districtwide 


performance levels in both mathematics and ELA. 


Table 7.5 


Baseline Characteristics of the Pooled Group of Treated Students Taught by Cohort 1 and 2 Elementary 
School Teachers Participating in LDC in 2 Consecutive Years and Comparison Students After Matching 


Treatment group 


Comparison group 


Student characteristic (n = 685) (n = 685) 

Race/ethnicity 

Hispanic (%) 72.6 72.6 

Black (%) 11.4 11.4 

Asian (%) 4.4 4.4 

White (%) 10.4 10.2 

Other (%) 1.2 1.4 
Female (%) 49.8 48.3 
Special programs status 

Poverty (%) 59.4 59.4 

English language learner (%) 15.0 15.0 

Special education (%) 6.3 6.3 

Gifted (%) 12.7 8.9 
Student baseline achievement 

Mean baseline year mathematics Z score 0.134 0.145 

Mean baseline year ELA Z score 0.143 0.153 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers 0.126 0.091 

Teacher years of experience 10.5 14.7 
Grade level at baseline year 

Grade 3 (%) 86.0 86.0 

Grade 4 (%) 14.0 14.0 


111 


The pooled LDC middle student sample (see Table 7.6) was almost entirely Hispanic and 
more than two thirds of students were eligible for free and reduced-priced lunches. English 
language learners represented one fifth of this sample, while special education students and 
those classified as gifted each represented about 10% of the sample. The sample was mostly 
composed of students in fifth and sixth grades (87.5% combined) in the baseline year. In 
addition, mean performance on the baseline year assessments was slightly lower for LDC 
students as compared to districtwide performance levels in mathematics and ELA. 
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Table 7.6 


Baseline Characteristics of the Pooled Group of Treated Students Taught by Cohort 1 and 2 Middle School 
Teachers Participating in LDC in 2 Consecutive Years and Comparison Students After Matching 


Treatment group Comparison group 
Student characteristic (n = 3,294) (n = 3,294) 

Race/ethnicity 

Hispanic (%) 94.7 94.7 

Black (%) 2.7 2.7 

Asian (%) 0.7 0.6 

White (%) 1.4 1.3 

Other (%) 0.5 0.7 
Female (%) 50.9 50.9 
Special programs status 

Poverty (%) 71.6 70.9 

English language learner (%) 19.2 19.2 

Special education (%) 7.7 8.4 

Gifted (%) 12.0 13.8 
Student baseline achievement 

Mean baseline year mathematics Z score -0.051 -0.050 

Mean baseline year ELA Z score -0.092 -0.099 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.101 -0.098 

Teacher years of experience 15.4 13.3 
Grade level at baseline year 

Grade 4 (%) 12.5 12.5 

Grade 5 (%) 47.1 47.1 

Grade 6 (%) 40.4 40.4 


Table 7.7 reports the sample information for the students taught by the final pooled LDC 
sample across both cohorts and school levels. As can be seen, this sample included a very large 
proportion of Hispanic students (90.9%), and a majority of students with low socioeconomic 
backgrounds (69.5%). English language learners represented slightly more than one fifth of this 
sample, while there were 12.1% special education students. The sample was mostly composed 
of students who were in fifth grade (39.0%) or sixth grade (33.4%) during the baseline year. In 
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addition, mean performance on the baseline year assessment was slightly lower for LDC 
students as compared to districtwide performance levels in mathematics and ELA. 


Table 7.7 
Baseline Characteristics of the Pooled Group of Treated Students Taught by Cohort 1 and 2 Elementary 
and Middle School Teachers Participating in LDC in 2 Consecutive Years and Comparison Students After 
Matching 


Treatment group Comparison group 
Student characteristic (n = 3,979) (n = 3,979) 

Race/ethnicity 

Hispanic (%) 90.9 90.9 

Black (%) 4.2 4.2 

Asian (%) 1.4 1.3 

White (%) 2.9 2.9 

Other (%) 0.6 0.7 
Female (%) 50.7 50.5 
Special programs status 

Poverty (%) 69.5 69.0 

English language learner (%) 18.5 18.5 

Special education (%) 7.5 8.1 

Gifted (%) 12.1 13.0 
Student baseline achievement 

Mean baseline year mathematics Z score -0.019 -0.017 

Mean baseline year ELA Z score -0.051 -0.055 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.062 -0.065 

Teacher years of experience 14.6 13.5 
Grade level at baseline year 

Grade 3 (%) 14.8 14.8 

Grade 4 (%) 12.8 12.8 

Grade 5 (%) 39.0 39.0 

Grade 6 (%) 33.4 33.4 


114 


Table 7.8 reports the sample information for the students taught by Cohort 1 middle 
school teachers with 3 years of LDC. This sample consisted almost entirely of Hispanic students 
(96.5%) and students with low socioeconomic backgrounds (95.4%). English language learners 
represented slightly more than one third of this sample, while there were 7.5% special 
education students and 10.1% gifted students. The sample was mostly composed of students 
that were in fourth grade (59.2%) or fifth grade (39.6%) in the baseline year. In addition, mean 
performance on the baseline year assessments was about one quarter of a standard deviation 
lower for LDC students as compared to districtwide performance levels in mathematics and 
ELA. 
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Table 7.8 


2015-2016 Baseline Characteristics of Treated Students Taught by Cohort 1 Middle School Teachers 
Participating in LDC in 3 Consecutive Years and Comparison Students After Matching 


Treatment group Comparison group 
Student characteristic (n = 801) (n = 801) 

Race/ethnicity 

Hispanic (%) 96.5 96.5 

Black (%) 3.0 1.5 

Asian (%) 0.0 0.1 

White (%) 0.0 0.2 

Other (%) 0.5 1.7 
Female (%) 
Special programs status 

Poverty (%) 95.4 95.4 

English language learner (%) 36.7 36.7 

Special education (%) 75 7.5 

Gifted (%) 10.1 7.9 
Student baseline achievement 

Mean baseline year mathematics Z score -0.202 -0.250 

Mean baseline year ELA Z score -0.257 -0.248 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.269 -0.222 

Teacher years of experience (<=3 years) 24.9 15.4 
Grade level at baseline year 

Grade 3 (%) 1.2 1.2 

Grade 4 (%) 59.2 59.2 

Grade 5 (%) 39.6 39.6 


7.3 Primary Outcome Analysis Results: Teachers Participating in LDC for 2 
Consecutive Years Across Cohorts 


Here, we examine the impact of LDC as practiced by teachers with 2 consecutive years of 
experience with the program. We first present the results for the elementary and middle 
schools separately, after which we present the results of an analysis pooling the two groups of 
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schools together. In the analytic sample, students were typically exposed to only one teacher at 
the elementary level. As discussed in detail in Chapter 2, when elementary students were 
exposed to more than one teacher, their exposure was weighted with a sum of 1.0. In contrast, 
middle school students could be exposed to between zero and six different LDC teachers in a 
given year. Using a MMMC design, each observation at Level 1 represented one student. 
Weights across teachers for each student summed to a unity (1). 


Two different approaches were used to model the LDC treatment intervention variable as 
a fixed effect at the student level. The first model was dosage dependent, and took into 
account variation in students’ level of exposure to LDC teachers. In this approach, the 
treatment variable was structured as a continuous response variable between zero and one, 
based on exposure to LDC teachers. In contrast, the second approach was modeled as dosage 
independent, and considered any student exposed to any LDC intervention teacher to be a 
treated individual. In this latter approach the treatment variable was dichotomous, coded as 
one for LDC treated students and zero for comparison students. 


In Tables 7.9 and 7.10 we present results of both the dosage-dependent and dosage- 
independent models, estimating the impact of LDC on Cohort 1 and Cohort 2 students in 
elementary and middle schools respectively. As can be seen, model results for the LDC effect on 
elementary student outcomes are in the positive direction, but are not statistically significant 
for either model. In other words, neither analysis provided sufficient evidence to conclude that 
elementary students taught by LDC teachers performed better on the ELA test than did their 
matched peers in the comparison group. As the analysis focused only on students in grades 5 
and 6 in the outcome year, it is not generalizable to the full population of elementary school 
teachers implementing LDC. 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Baseline ELA performance was the strongest 
predictor and baseline mathematics performance also helped explain the outcome. In addition 
to baseline achievement, three student-level demographic variables helped predict 
performance: English language learners performed at lower levels than English only and 
reclassified fluent English proficient students, females performed at significantly higher levels 
than males, and Black students performed at lower levels than did White students. 
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Table 7.9 


Effect Estimates of Cohorts 1 and 2 Elementary School Teachers Participating in LDC in 2 Consecutive 
Years on Smarter Balanced ELA Performance, Dosage-Dependent and Dosage-Independent Models 


Variable 


Dosage-dependent model 


coefficient (SD) 


Dosage-independent model 


coefficient (SD) 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 
Black 
Poverty 
Female 
English language learner 
Special education 
Gifted 
Teacher experience 
Baseline peer ELA Z score 
Baseline year mathematics Z score 
Baseline year ELA Z score 
Grade 3 at baseline 


Cohort 2 schools 


0.028 (0.078) 


-0.095 (0.057) 
-0.236 (0.081)* 
-0.020 (0.036) 
0.118 (0.032)* 
-0.106 (0.051)* 
-0.025 (0.068) 
0.025 (0.057) 
-0.166 (0.075)* 
0.154 (0.053)* 
0.265 (0.030)* 
0.542 (0.029)* 
-0.103 (0.083) 
-0.106 (0.121) 


0.021 (0.078) 


-0.095 (0.057) 
-0.236 (0.081)* 
-0.020 (0.036) 
0.118 (0.032)* 
-0.106 (0.051)* 
-0.025 (0.068) 
0.025 (0.057) 
-0.166 (0.075)* 
0.154 (0.053)* 
0.265 (0.030)* 
0.542 (0.029)* 
-0.103 (0.083) 
-0.106 (0.121) 


Note. Based on the dosage-dependent model, the average treated student received a 0.997 treatment 


dosage. Because of this, using the dosage-dependent model we could estimate the ATT at 


(0.997 x 0.028) = 0.028. 
*C/l two-tailed probability > .95 


As presented in Table 7.10, dosage-dependent model results indicate a statistically 


significant and positive LDC effect on the ELA scores of the pooled sample of Cohort 1 and 
Cohort 2 middle school students. In contrast, the dosage-independent model did not yield a 
statistically significant LDC effect (it bears noting that LDC’s preferred model was dosage- 
dependent, cross-disciplinary teacher team treatments). The significant dosage-dependent 
effect suggests that increased student exposure to LDC teachers was connected to improved 
ELA test performance. More specifically, treatment students with exposure to LDC in all three 
subjects were estimated to perform 0.281 standard deviations above matched comparison 
students. The average treatment student, however, had a 43.3% exposure to LDC in core 
content classes, and is estimated to perform 0.122 standard deviations above matched 
comparison students, the average treatment on treated effect (ATT). 
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Table 7.10 


Effect Estimates of Cohorts 1 and 2 Middle School Teachers Participating in LDC in 2 Consecutive Years on 


Smarter Balanced ELA Performance, Dosage-Dependent and Dosage-Independent Models 


Variable 


Dosage-dependent model 


coefficient (SD) 


Dosage-independent model 


coefficient (SD) 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 

Black 

Poverty 

Female 

English language learner 
Special education 

Gifted 

Honors English 

Teacher experience 


Baseline peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 
Grade 4 at baseline 
Grade 5 at baseline 


Cohort 2 schools 


0.281 (0.081) * 


-0.201 (0.047) * 
-0.369 (0.064) * 
-0.050 (0.017)* 
0.180 (0.014)* 
-0.130 (0.022)* 
0.036 (0.029) 

0.048 (0.024)* 
0.134 (0.035)* 
-0.031 (0.058) 

0.178 (0.037)* 
0.200 (0.013)* 
0.533 (0.014)* 
0.006 (0.041) 

-0.075 (0.024)* 
0.018 (0.029) 


0.101 (0.054) 


-0.202 (0.047)* 
-0.370 (0.064)* 
-0.050 (0.017)* 
0.180 (0.014)* 
-0.130 (0.022)* 
0.034 (0.029) 
0.047 (0.024) 
0.133 (0.035)* 
-0.031 (0.059) 
0.179 (0.037)* 
0.200 (0.013)* 
0.534 (0.014)* 
0.007 (0.041) 
-0.069 (0.024)* 
0.014 (0.029) 


Note. Based on the dosage-dependent model, the average treated student received a 0.433 treatment 
dosage. Because of this, using the dosage-dependent model we could estimate the ATT at 

(0.433 x 0.281) = 0.122. 

*CI two-tailed probability > .95 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Baseline ELA performance remained the 
strongest predictor and baseline mathematics performance also helped explain the outcome. In 
addition to baseline achievement, Hispanic students, Black students, students receiving free or 
reduced-price lunch, and English language learner status were all significant predictors of ELA 
performance and were in the expected directions. Students enrolled in honors English courses 
performed at higher levels than did their peers taking standard English courses. Students 
performed similarly in Cohort 1 and Cohort 2 schools. 
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Table 7.11 presents the results of models combining the two preceding samples of 
elementary and middle school students. As with the middle school model, the dosage- 
dependent model here indicates a statistically significant and positive LDC effect on the ELA 
scores of the combined sample of elementary and middle school students. The dosage- 
independent model coefficient was positive but not statistically significant. The significant 
dosage-dependent effect suggests that increased student exposure to LDC teachers was 
connected to improved ELA test performance. More specifically, treatment students with 
exposure to LDC were estimated to perform 0.124 standard deviations above matched 
comparison students. The average treatment student, however, had a 53.1% exposure to LDC 
in core content classes, and was estimated to perform 0.066 standard deviations above 
matched comparison students (ATT effect). 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Again, baseline ELA performance remained 
the strongest predictor and baseline mathematics performance also helped explain the 
outcome. In addition to baseline achievement, Hispanic students, students receiving free or 
reduced-price lunch, and English language learner status were all significant predictors of ELA 
performance and were in the expected directions. Students enrolled in honors English courses 
performed at higher levels than did their peers taking standard English courses, and female 
students performed significantly higher. Students performed similarly in Cohorts 1 and 2 
schools. 
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Table 7.11 


Effect Estimates of Cohorts 1 and 2 Elementary and Middle School Teachers Participating in LDC in 2 
Consecutive Years on Smarter Balanced ELA Performance, Dosage-Dependent and Dosage-Independent 
Models 


Dosage-dependent model Dosage-independent model 


Variable 


coefficient (SD) 


coefficient (SD) 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 
Black 
Poverty 
Female 
English language learner 
Special education 
Gifted 
Honors English 
Teacher experience 
Baseline peer ELA Z score 
Baseline year mathematics Z score 
Baseline year ELA Z score 
Grade 3 baseline 
Grade 4 baseline 
Grade 5 baseline 


Cohort 2 schools 


0.124 (0.054)* 


-0.154 (0.035)* 
-0.298 (0.048) 
-0.046 (0.015)* 
0.169 (0.013)* 
-0.125 (0.020)* 
0.026 (0.027) 
0.044 (0.022)* 
0.134 (0.030)* 
-0.078 (0.046) 
0.170 (0.030)* 
0.211 (0.012)* 
0.533 (0.012)* 
-0.011 (0.047) 
0.026 (0.037) 
-0.069 (0.024)* 
0.010 (0.028)* 


0.062 (0.046) 


-0.153 (0.036)* 
-0.297 (0.048) 
-0.046 (0.015)* 
0.169 (0.013)* 
-0.125 (0.020)* 
0.026 (0.027) 
0.045 (0.022)* 
0.132 (0.030)* 
-0.079 (0.046) 
0.172 (0.030)* 
0.210 (0.012)* 
0.533 (0.012)* 
0.002 (0.046) 
0.028 (0.038) 
-0.066 (0.024)* 
0.010 (0.028)* 


Note. Based on the dosage-dependent model, the average treated student received a 0.531 treatment 
dosage. Because of this, using the dosage-dependent model we could estimate the ATT at 

(0.531 x 0.124) = 0.066. 

*C/l two-tailed probability > .95 


7.4 Primary Outcome Analysis Results: Cohort 1 Teachers Participating in LDC 
for 3 Consecutive Years 


In this section, we examine the impact of LDC as practiced by middle school teachers with 
3 consecutive years of experience with the program. As can be seen in Table 7.12, model results 
for the LDC effect on student outcomes were in the positive direction, but were not statistically 
significant for either model. In other words, neither analysis provided sufficient evidence to 
conclude that middle school students taught by LDC teachers performed better on the ELA test 
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than did their matched peers in the comparison group. However, the diminished sample size via 
attrition among the Cohort 1 teachers likely influenced the model’s ability to detect an effect. 


The significant effects of the covariates on student performance were similar under both 
models and were in the expected direction. Once again, baseline ELA performance was the 
strongest predictor and baseline mathematics performance also helped explain the outcome. In 
addition, students in honors English courses performed at higher levels than did their peers 
taking standard English courses, and female students performed significantly higher. 


Table 7.12 


Effect Estimates of Cohort 1 Middle School Teachers Participating in LDC in 3 Consecutive Years on 
Smarter Balanced ELA Performance, Dosage-Dependent and Dosage-Independent Models 


Dosage-dependent model Dosage-independent model 


Variable 


coefficient (SD) 


coefficient (SD) 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 
Black 
Poverty 
Female 
English language learner 
Special education 
Gifted 
Honors English 
Teacher experience 
Baseline peer ELA Z score 
Baseline year mathematics Z score 
Baseline year ELA Z score 
Grade 3 at baseline 


Grade 4 at baseline 


0.148 (0.233) 


-0.069 (0.144) 
-0.001 (0.177) 
0.013 (0.076) 
0.234 (0.033)* 
-0.077 (0.040) 
-0.124 (0.071) 
0.081 (0.063) 
0.200 (0.082)* 
0.070 (0.122) 
0.222 (0.083)* 
0.218 (0.030)* 
0.509 (0.030)* 
-0.185 (0.170) 
0.076 (0.047) 


0.123 (0.127) 


-0.069 (0.144) 
-0.003 (0.177) 
0.013 (0.076) 
0.234 (0.033)* 
-0.077 (0.040) 
-0.120 (0.071) 
0.081 (0.063) 
0.197 (0.082)* 
0.050 (0.120) 
0.225 (0.084)* 
0.218 (0.030)* 
0.509 (0.030)* 
-0.193 (0.171) 
0.075 (0.046) 


Note. Based on the dosage-dependent model, the average treated student received a 0.318 treatment 


dosage. Because of this, using the dosage-dependent model we could estimate an ATT at 


(0.318 x 0.148) = 0.047. 
*Cl two-tailed probability > .95 
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7.5 Supplementary Outcome Analysis Results: Cohort 2 Teachers Participating in 
LDC for 2 Consecutive Years 


In this section, we present results of both the dosage-dependent and dosage-independent 
models on Cohort 2 elementary and middle school students’ ELA performance in 2018-2019 
after being taught by teachers with 2 years of LDC experience, respectively. As can be seen in 
Table 7.13, model results for the LDC effect on elementary student outcomes are in the positive 
direction, but are not statistically significant for either model. In other words, neither analysis 
provided sufficient evidence to conclude that elementary students taught by LDC teachers 
performed better on the ELA test than did their matched peers in the comparison group. 


Table 7.13 


Effect Estimates of Cohort 2 Elementary School Teachers Participating in LDC in 2 Consecutive Years on 
2018-2019 Smarter Balanced ELA Performance, Dosage-Dependent and Dosage-Independent Models 


Dosage-dependent model Dosage-independent model 


Variable 


coefficient (SD) 


coefficient (SD) 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 
Black 
Poverty 
Female 
English language learner 
Special education 
Gifted 
Teacher experience 
Baseline peer ELA Z score 
Baseline year mathematics Z score 
Baseline year ELA Z score 


Grade 3 at baseline 


0.009 (0.087) 


-0.105 (0.058) 
-0.222 (0.085) * 
-0.023 (0.036) 
0.118 (0.033)* 
-0.101 (0.052) 
-0.025 (0.068) 
0.035 (0.059) 
-0.176 (0.076)* 
0.144 (0.054)* 
0.262 (0.031)* 
0.543 (0.030)* 
-0.110 (0.083) 


0.007 (0.078) 


-0.105 (0.058) 
-0.222 (0.085) * 
-0.023 (0.036) 
0.118 (0.033)* 
-0.101 (0.052) 
-0.025 (0.068) 
0.035 (0.059) 
-0.176 (0.076)* 
0.144 (0.054)* 
0.262 (0.031)* 
0.543 (0.030)* 
-0.110 (0.083) 


Note. Based on the dosage-dependent model, the average treated student received a 0.997 treatment 


dosage. Because of this, using the dosage-dependent model we could estimate an ATT at 


(0.997 x 0.009) = 0.009. 
*C/l two-tailed probability > .95 


The significant effects of the covariates on student performance also were similar under 


the two models and were in the expected direction. Baseline ELA performance was the 


strongest predictor and baseline mathematics performance also helped explain the outcome. In 
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addition to baseline achievement, two demographic variables helped predict performance: 
females performed at significantly higher levels than males, and Black students performed at 
lower levels than did White students. 


The results for the Cohort 2 middle school students taught by teachers with 2 years of LDC 
experience tell a different story (Table 7.14). Both the dosage-dependent and dosage- 
independent model results indicate a statistically significant and positive LDC effect on the 
middle school student outcome. The significant dosage-dependent effect suggests that 
increased student exposure to LDC teachers was connected to improved ELA test performance. 
More specifically, treatment students with exposure to LDC in all three subjects were estimated 
to perform 0.367 standard deviations above matched comparison students. The average 
treatment student, however, had a 41.2% exposure to LDC in core content classes, and was 
estimated to perform 0.139 standard deviations above matched comparison students (the ATT 
effect). Note that we reported the effect of Cohort 1 middle school teachers in their second 
year of implementation in our prior report (Wang et al., 2019); those results did not find a 
statistically significant impact. When both cohorts were combined, however, an impact was 
found (see Tables 7.10 and 7.11). As such, it appears that the Cohort 2 middle schools were the 
main driver of positive impact. 
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Table 7.14 


Effect Estimates of Cohort 2 Middle School Teachers Participating in LDC in 2 Consecutive Years on 2018- 


2019 Smarter Balanced ELA Performance, Dosage-Dependent and Dosage-Independent Models 


Variable 


Dosage-dependent model 


coefficient (SD) 


Dosage-independent model 


coefficient (SD) 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 
Black 
Poverty 
Female 
English language learner 
Special education 
Gifted 
Honors English 
Teacher experience 
Baseline peer ELA Z score 
Baseline year mathematics Z score 
Baseline year ELA Z score 
In Grade 4 at baseline 


In Grade 5 at baseline 


0.367 (0.104)* 


-0.203 (0.049)* 
-0.405 (0.071)* 
-0.044 (0.018)* 
0.168 (0.017)* 
-0.125 (0.028)* 
0.001 (0.038) 

0.060 (0.028)* 
0.116 (0.042)* 
-0.001 (0.067) 

0.204 (0.044)* 
0.190 (0.016)* 
0.543 (0.017)* 
0.024 (0.046) 

-0.064 (0.028)* 


0.131 (0.068)* 


-0.204 (0.050) * 
-0.407 (0.071) * 
-0.044 (0.018)* 
0.168 (0.017)* 
-0.126 (0.028)* 
0.000 (0.038) 
0.060 (0.028)* 
0.113 (0.043)* 
0.004 (0.068) 
0.205 (0.045)* 
0.189 (0.016)* 
0.543 (0.017)* 
0.027 (0.046) 
-0.056 (0.028)* 


Note. Based on the dosage-dependent model, the average treated student received a 0.412 treatment 
dosage. Because of this, using the dosage-dependent model we could estimate an ATT at 

(0.412 x 0.338) = 0.139. 

*C/l two-tailed probability > .95 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected direction. Baseline ELA performance was the strongest 
predictor and baseline mathematics performance also helped explain the outcome. In addition 
to baseline achievement, a number of demographic characteristics helped predict the outcome. 
Hispanic students, Black students, students receiving free or reduced-price lunch, and English 
language learners performed at lower levels than their reference groups. In contrast, females, 
gifted students, and students enrolled in honors English courses performed at higher levels. 
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7.6 Supplementary Outcome Analysis Results: Prior Year and Outcome Year 
Exposure Subgroups 


In this section, we present results exploring subgroups of teachers and students from the 
Cohorts 1 and 2 middle school sample. These analyses explore whether the positive results 
found for the overall Cohorts 1 and 2 middle school sample are being driven by subgroups of 
teachers and/or students. 


First, we explore whether students with sustained exposure to LDC over time performed 
at relatively higher levels than students exposed to LDC for the first time in the primary 
outcome year. To answer this question, we constructed two mutually exclusive subgroups of 
students: students who were exposed to LDC in 2 consecutive years (2016-2017 and 2017- 
2018 for Cohort 1; 2017-2018 and 2018-2019 for Cohort 2), and students who were only 
exposed to LDC in the second year (2017-2018 for Cohort 1; 2018-2019 for Cohort 2). Matched 
control groups were selected for each subgroup of students. The full results of the model can 
be found in Table I7 in Appendix |. The model includes indicator variables for the two treatment 
groups and the matched control group for the students exposed to LDC in 2 consecutive years. 
The control group for the students exposed to LDC in just the outcome year was used as the 
reference group in the model. 


Our primary substantive interest, however, is not in the coefficients as displayed in 
Table 17, but rather in two post hoc tests of the statistical difference between coefficients, 
displayed in Table 7.15. Using the MLwiN software,’ we obtained the chi square value (1 degree 
of freedom) for a test of the difference between two coefficients. The first test examines the 
difference between the coefficient for the treatment group of students exposed to LDC in 2 
consecutive years, and the coefficient for the control group selected specifically for these 
treatment students. That test shows a difference of 0.102. Although the difference is in the 
expected direction, with the treatment group coefficient higher than the control group 
coefficient, the difference (with a p value of .076) is not significant. The second test examines 
the difference between the two treatment groups: those students with 2 years of LDC exposure 
and those students with only exposure in the outcome year. Again, the difference is in the 
expected direction but not significant (p = .097). 


“See http://www. bristol.ac.uk/cmm/software/mlwin/ 


126 


Table 7.15 
Subgroup Results Examining Exposure of Students to LDC Over Time 


Difference test Difference xX? p value 


Difference between coefficient for students exposed to 0.102 2.753 0.076 
LDC for 2 consecutive years and coefficient for matched 
control student group 


Difference between coefficient for students exposed to 0.045 4.505 0.097 
LDC for 2 consecutive years and coefficient for students 
with exposure to LDC only in the outcome year 


As described earlier, our analysis of LDC as practiced by middle school teachers with 2 
years of participation showed a positive impact in both the dosage-dependent and dosage- 
independent models. Next we explore whether the impact for students with greater exposure 
to LDC teachers is different from the impact for students with less exposure to LDC teachers. 
Again, we constructed two mutually exclusive treatment groups: a high-dosage group consisting 
of students who were exposed to LDC in half or more of their core content area class time, and 
a low-dosage group consisting of students who were exposed to LDC in less than half of their 
core content area class time. Like the model for which results are displayed in Table 7.15 and 
Table I7 in Appendix I, this analysis selected matched control groups for each of the two 
mutually exclusive treatment groups, and indicators for three of the four groups were included 
in the model. Full model results are displayed in Table 18 in Appendix |, and the difference tests 
of interest are displayed in Table 7.16 below. 


Table 7.16 
Subgroup Results Examining High and Low LDC Dosage in the Outcome Year 


Difference test Difference x’ p value 


Difference between coefficient for students exposed to 0.142* 5.138 0.023 
LDC in over half of their core content class time and 
coefficient for matched control student group 


Difference between coefficient for students exposed to 0.099* 4.505 0.034 
LDC in over half of their core content class time and 

coefficient for students exposed to LDC in less than half 

of their core content class time 


As can be seen the coefficient for the high-dosage treatment group is larger than the 
coefficients for both the high-dosage matched control group and the low-dosage treatment 
group, and these differences (0.142 and 0.099 respectively) are statistically significant. These 
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results suggest that the overall positive impact of LDC as practiced by middle school teachers in 
their second consecutive year of implementation is disproportionately driven by the 
performance of students exposed to LDC teachers in greater than half of their core content 
class time. 


7.7 Summary and Interpretation of Results 


Here we summarize the student outcome analysis results. Table 7.17 includes the results 
of all of our analyses of the impact of LDC as practiced by teachers with 2 or more years of 
experience with program, including both analyses conducted this year and last year. In our 
previous report (Wang et al., 2019), we shared results of the impact of Cohort 1 middle school 
teachers with 2 years of LDC participation. That analysis did not identify a statistically significant 
effect, and a parallel analysis on Cohort 1 elementary school teachers was not possible due to 
insufficient sample size. In this report, we share for the first time the results of the impact of 
Cohort 2 teachers alone and the pooled group of Cohort 1 and Cohort 2 teachers. We also share 
the results of an analysis looking at the impact of Cohort 1 middle school teachers with 3 
consecutive years of LDC experience. 
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Table 7.17 


Impact of LDC on State ELA Assessment Scores by Cohort and School Level 


Cohort/teachers 


LDC Analytical Sample 
Information 


Dosage- 
dependent 
model 
coefficient (SD) 


Dosage- 
independent 
model 
coefficient (SD) 


Cohort 1 elementary 
schools/teachers with 2 years of LDC 


Cohort 1 middle schools/teachers 
with 2 years of LDC” 


Cohort 2 elementary 
schools/teachers with 2 years of LDC 


Cohort 2 middle schools/teachers 
with 2 years of LDC 


Cohorts 1 and 2 elementary 
schools/teachers with 2 years of LDC 


Cohorts 1 and 2 middle 
schools/teachers with 2 years of LDC 


Cohorts 1 and 2 elementary and 
middle schools/teachers with 2 
years of LDC 


Cohort 1 middle schools/teachers 
with 3 years of LDC 


2 schools / 3 teachers / 
45 students 


4 schools / 22 teachers / 
995 students 


15 schools / 31 teachers 
/ 640 students 


7 schools / 32 teachers / 
2,299 students 


17 schools / 34 teachers 
/ 685 students 


11 schools / 54 teachers 
/ 3,294 students 


26 schools / 89 teachers 
/ 3,979 students 


3 schools / 8 teachers / 
801 students 


Not conducted 


0.118 (0.116) 


0.009 (0.087) 


0.367 (0.104)* 


0.028 (0.078) 


0.281 (0.081)* 


0.124 (0.054)* 


0.148 (0.233) 


Not conducted 


0.050 (0.083) 


0.007 (0.078) 


0.131 (0.068)* 


0.021 (0.078) 


0.101 (0.054) 


0.062 (0.046) 


0.123 (0.127) 


*Full results can be found in our earlier report: Wang et al., 2019. 


*Significant at p < .05. 


We found statistically significant positive effects of Cohort 2 middle school teachers with 
2 years of LDC implementation using both the dosage-dependent and dosage-independent 
treatment models. Cohort 2 middle school teachers and students formed the greatest part of 


our pooled samples, and as a result, the dosage-dependent positive effect also carried over to 
our analyses testing the impact of LDC as practiced for 2 consecutive years by all middle school 
teachers (Cohorts 1 and 2), and by all elementary and middle school teachers (again including 
Cohorts 1 and 2). These effects were not detected using the dosage-independent version of the 
treatment variable. An impact at the elementary level alone was not found, although it is 
important to note that the teacher and student samples for the elementary analyses were 
considerably smaller than for the middle school analysis, and the analyses only account for 
elementary teachers and students at Grades 5 and 6 in the outcome year. Our analysis of the 
impact of Cohort 1 middle school teachers after 3 years of participation also did not show a 
statistically significant impact, although the coefficient was positive. As noted in Chapter 2, 
there was substantial attrition in Cohort 1. The sample of teachers who began implementing in 
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2016-2017 and continued through 2018-2019 was limited to eight teachers and 801 students, 
which was not large enough to detect an impact based on our earlier power analysis. 


Next we provide a lens through which the reader can contextualize the magnitude of the 
results. We present dosage-dependent effects for each of the three primary analyses based on 
teachers with 2 years of LDC implementation. Figure 7.1 depicts the estimated impacts of LDC 
in the three samples of students exposed to LDC teachers in all three major content areas: ELA, 
social studies/history, and science. These effect sizes can be best understood as the estimated 
impact of LDC under ideal conditions. Figure 7.2 depicts the estimated impact of LDC in the 
three samples on the average observed student, who in the middle school context had 
considerably less exposure to LDC teachers in his or her core content classes. 


Figure 7.1 


Treatment Effect on Smarter Balanced ELA Scores With 95% Credibility Interval for Students With Full 
LDC Dosage, by Cohort 


0.50 - 
0.25 
Teacher Cohorts 
= Elementary Teachers (n=36) 
0.00--- 
| Middle Teachers (n=54) 
3 Elementary and Middle Teachers (n=90) 
-0.25- 
-0.50- 
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Figure 7.2 


Treatment Effect on Smarter Balanced ELA Scores With 95% Credibility Interval for Students With 
Average LDC Dosage, by Cohort 


0.50 - 


0.25 - 
Teacher Cohorts 
ae = Elementary Teachers (n=36) 
00 fae 
HHI Middle Teachers (n=54) 
Sj Elementary and Middle Teachers (n=90) 
0.25 


The figures make clear that middle school teachers were the primary driver behind the 
significant positive effects. Both the elementary and middle school coefficients were in the 
positive direction, but the magnitude of the middle school coefficient was about 10 times larger 
when looking at the impact of full LDC dosage. As a result, the lower bound of the credibility 
interval for the middle school coefficient is well above the zero line, whereas the lower bound 
of the credibility interval for the elementary school coefficient is substantially below the zero 
line. When the two samples are combined for one analysis, the lower bound of the credibility 
interval is just above the zero line. Thus, evidence of differential impacts across school levels is 
strong, with a clear impact at the middle school level and no evidence of an impact at the 
elementary school level. It is important again to note that elementary analyses relied ona 
smaller teacher and student sample than the middle school analyses, and only included 
teachers and students in Grades 5 and 6 in the outcome year. 


To help the reader contextualize the statistically significant effect for middle school 
teachers, we utilize an approach developed by Hill et al. (2008), which involves benchmarking 
against average student gains over the course of a school year. The authors reviewed annual 
achievement gains in seven nationally normed reading assessments: CAT5, SAT9, Terra Nova- 
CTBS, Gates-MacGinitie, MAT8, Terra Nova-CAT, and SAT10. They found that students gained 
an average of .32 standard deviations from Grades 5 to 6, .23 standard deviations from Grades 
6 to 7, and .26 standard deviations from Grades 7 to 8. A simple mean of these three average 
gains is .27. 


131 


Using this benchmark, and assuming a 9-month school year, the .122 effect estimate for 
students with average observed LDC dosage is of similar magnitude to 4.1 months of learning 
in the Hill et al. (2008) meta-analysis [(.122/.27)*9=4.1]. Likewise, the .281 effect estimate for 
students with full LDC dosage aligns to approximately 9.4 months of schooling 
[(.281/.27)*9=9.4]. It is important to note again that the ideal conditions of students being 
exposed to LDC in all three core content areas across the whole school year was not met for 
most students; therefore the extrapolation of 4.1 months is the figure best aligned with the 
actual observed effect of LDC. 
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8.0 Summary of Findings 


This final annual report examines LDC following 3 years of implementation in Cohort 1 
schools and 2 years of implementation in Cohort 2 schools. We summarize these results 
organized by the three categories of evaluation questions listed in Chapter 1 of this report. 


8.1 Program Characteristics and Implementation 


The study schools serve largely Hispanic populations, with a high proportion of students 
qualifying for free and reduced-price lunch, and many English language learners. Consistent 
with prior year results, participants reported overwhelmingly positive attitudes toward LDC and 
its implementation at their schools. Teachers generally appreciated the opportunity to 
collaborate with colleagues, and nearly uniformly praised their LDC coaches. 


Evidence suggests that implementation was largely faithful to the intended structure of 
LDC, although the level of implementation was not uniformly high. The frequency of PLC 
meetings varied greatly across schools, with some PLCs struggling to meet frequently and 
ensure high attendance. On a positive note, PLC members were nearly uniform in reporting that 
their teacher leaders were supportive, knowledgeable, and helpful. Teacher leaders themselves 
also reported high satisfaction with support from coaches, professional development offerings, 
and how the teacher leader role allowed them to be instructional leaders in their schools. 


Analysis of program data suggests that nearly all participants were engaging with the 
module building platform at least to some degree, but that engagement varied considerably. 
About half of teachers failed to engage in module design at a basic level by editing the teaching 
task, while others engaged deeply by editing multiple module elements. When coaches 
provided feedback via commenting on modules and other methods, PLC members generally 
found it useful. But our fidelity analysis revealed that 40% of modules did not receive the 
program goal of two coach comments. Furthermore, the national peer review process was used 
by few participants. Module analysis suggests that the materials adapted and created by PLC 
members varied in levels of completion, and based on the presence of uploaded student work, 
a sizable number of modules might not have been implemented in the classroom. 


8.2 Contextual Factors and Implementation 


Coaches and teacher leaders were almost universally praised by teachers participating in 
LDC. While teacher respondents generally reported that their school administrators were 
supportive of the program, administrators’ level of participation in PLC meetings and their 
observation of LDC instruction varied greatly across the sample. There was substantial attrition 
of schools and teachers from the program across the 3 years, with many school leaders 
deciding that they didn’t have sufficient resources and/or teacher buy-in to sustain the 
program. Most schools that remained in the program at the end of the study, however, 
indicated that they would continue to use LDC practices and tools, at least to some extent. 
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While school personnel generally perceived district leaders as supportive of LDC, some 
educators didn’t feel district leaders fully understood the program. 


8.3 Program Impacts 


Based on survey results, LDC was perceived to have positive impacts on a range of both 
teacher practices and student skills. Teachers, in general, felt that LDC had improved their 
instructional planning and pedagogical skill sets and was also helping to promote collaboration 
between teachers. Teachers and administrators also felt that LDC was improving student 
learning across multiple domains, in particular writing skills. Furthermore, those attitudes 
around teacher and student skills became even more positive over time on average for teachers 
completing the survey in both 2017-2018 and 2018-2019. Analysis of module quality showed 
improvement from 2016-2017 to 2017-2018, but average quality seemed to decrease slightly 
from 2017-2018 to 2018-2019. 


Quasi-experimental analyses demonstrated a positive impact of LDC as implemented by 
middle school teachers with 2 years of LDC experience on student ELA scores. Results using a 
dosage-dependent treatment variable suggest that students who were exposed to a greater 
amount of LDC instruction (via multiple participating teachers in different content areas) 
benefited more from the program. An analysis breaking up treatment students into two groups 
based on the dosage of LDC instruction they received substantiated the finding that impact was 
greater for those students exposed to LDC in more classes. In contrast, a statistically significant 
impact was not found at the elementary school level, where students were typically exposed to 
just one LDC teacher, and one to two LDC modules. 


8.4 Overall Conclusions 


Our mixed methods evaluation of LDC provides impressive evidence of the effectiveness 
of LDC as a tool to engage teachers around improving literacy instruction and to improve 
student learning around reading and writing. At its core, LDC provides a structure for teachers 
to meet, share practices, and collaboratively learn about how to infuse reading and writing into 
their curriculum and implement instruction that gives students opportunities to learn content 
and work on their reading and writing skills simultaneously. 


This study provides ample evidence that teachers valued the opportunity to learn 
together, and appreciated the range of in-person and digital supports that LDC offered. The 
time commitment was substantial, and not all teachers, administrators, and schools were able 
to sustain that commitment in the face of competing demands for attention and time. But 
those educators that remained in the program felt that it had a measurable impact on the skills 
of both teachers and students. That perception was confirmed at the middle school level by our 
rigorous quasi-experimental analyses, which show a substantial positive impact of LDC as 
practiced by middle school teachers with 2 years of LDC experience. Furthermore, middle 
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school students exposed to multiple LDC teachers across different content areas (ELA, social 
studies, and science) benefited more from the program than students with less exposure. 


Ensuring administrator and teacher buy-in, and dedicating substantial resources to the 
program (common planning time in particular) would likely yield even more positive results. 
Investment in further improvements to the support and delivery model at the elementary 
school level may also widen LDC’s impact. 
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Appendix A: 
Teacher Survey and Responses 


2018-2019 
LDC Teacher Survey 


LDC Participation 


T1. Please select your school from the drop down box. 
Teachers are skipped to T3 if they teach in an elementary school. 


T2a. In the current school year (2018-19), how many classes did you teach? 


| | classes n = 82, Mean = 4.78, Range: 1-7 


T2b. In how many of these classes did you use LDC modules and/or mini-tasks? 


| | classes n = 82, Mean = 2.86, Range: 1-6 


T2c. In what content areas did you use LDC modules and/or mini-tasks? 


T2d. In what grades did you use LDC modules and/or mini-tasks? 


T3. Prior to the current school year (2018-19), did you have any experience with LDC? 


(n=231) 
Yes 151 teachers (65.4%) 
No y Skip to T5a 80 teachers (34.6%) 
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T4. How many of the following did you teach prior to the current school year (2018-19)? 


| | LDC modules n= 151, Mean = 2.19, Range: 0-10 


| | LDC mini-tasks, outside of modules =n = 151, Mean = 2.78, Range: 0-20 


Professional Learning Community and Teacher Collaboration 


T5a. Did you participate this year in a Professional Learning Community (PLC) at least partly 
focused on implementing LDC in your school? 


(n = 199) 
Yes y Skip to T6 192 teachers (96.5%) 
No 7 teachers (3.5%) 


T5b. Did you use any LDC tools in your instructional planning or classroom instruction this 


year? 
(n=7) 
Yes GF Sip tosd 6 teachers (85.7%) 
No 1 teacher (14.3%) 


T5c. Why did you choose not to use any LDC tools in your instructional planning or 
classroom instruction this year? 


[Survey ends here for respondents answering question T5c] 
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T5d. What LDC tools did you use during the current school year? Select all that apply. 


LDC online courses 


Other (please specify) 


CoreTools online platform to access existing modules or mini-tasks 


CoreTools online platform to design modules or mini-tasks 


Modules or mini-tasks given to me by other teachers in my school 


[Survey ends here for respondents answering question T5d] 


T6. About how often did your LDC PLC meet? 


Less than once a month 


Once a month 


Every other week 


Once a week 


GY} XKip to T8 
FY} skip to T8 


Twice a week or more often 


(n = 224) 

12 teachers (5.4%) 
69 teachers (30.8%) 
118 teachers (52.7%) 
21 teachers (9.4%) 

4 teachers (1.8%) 


(n = 6) 

5 teachers (83.3%) 
5 teachers (83.3%) 
3 teachers (50.0%) 
4 teachers (66.7%) 


0 teachers 


T7. What were the primary barriers preventing your LDC PLC from meeting weekly? Select 


all that apply. 


PLC time was not protected. 


School administrator did not make it a priority. 


Not enough teachers participated. 


Other (please specify) 


PLC members had limited interest in attending meetings. 


Teacher Leader did not provide sufficient organizational support. 


PLC members had other priorities that compete with LDC participation. 


(n = 199) 


38 teachers (16.5%) 
25 teachers (10.8%) 
18 teachers (7.8%) 
1 teacher (0.4%) 

9 teachers (3.9%) 
99 teachers (42.9%) 
67 teachers (29.0%) 


T8. About how often did you have informal discussions (as opposed to scheduled meetings) 
about LDC with teachers in your LDC PLC? 


(n = 224) 
Less than once a month 20 teachers (8.9%) 
Once a month 51 teachers (22.8%) 
Every other week 62 teachers (27.7%) 
Once a week 75 teachers (33.5%) 
Twice a week or more 16 teachers (7.1%) 


T9. On average, how long did your school’s LDC PLC meetings typically last? 


(n = 224) 
Less than 45 minutes 12 teachers (5.4%) 
45 to 59 minutes 148 teachers (66.1%) 
60 to 74 minutes 59 teachers (26.3%) 
75 minutes or more 5 teachers (2.2%) 


Teacher Training and Support 


T10. How effective was your LDC PLC in the following areas? 


Not A little Moderately Very 
effective effective effective effective 

Creating an environment in which teachers were 3 16 80 125 
comfortable working together (n = 224) (1.3%) (7.1%) (35.7%) (55.8%) 

ne ip ae : . es wee 
: (1.3%) (7.6%) (37.1%) (54.0%) 

(n = 224) 

Allowing space to share student work (n = 224) = eed 
oon ~ (1.3%) (10.3%) (40.6%) (47.8%) 

Helping teachers to improve their LDC 2 23 92 107 
instructional plans. (n = 224) (0.9%) (10.3%) (41.1%) (47.8%) 
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T11. How would you rate each of the following aspects of the online course material (in the 
Learn tab in LDC CoreTools) that your coach used or directed you to use? 


Poor Fair Good Excellent 

Clarity of information presented (n = 224) ° a3 aa a 
7 (2.7%) (14.7%) (49.1%) (33.5%) 

Relevance of information presented (n = 224) si 26 =f B2 
(1.8%) (12.5%) (49.1%) (36.6%) 

Ease of use (n =224) uy a2 ad a 
(7.6%) (23.2%) (43.3%) (25.9%) 

Fall Gh fey Scene vane te) : ss of a8 
(n= 224) : 7 (4.0%) (14.3%) (47.8%) (33.9%) 

Usefulness of videos (n = 224) a8 a6 nS a7 

~ . ‘0 . ‘0 . ‘0 . ‘0 

(5.8%) (25.9%) (47.3%) (21.0%) 

Degree to which course material helped teachers 7 38 110 69 
to create and/or adapt LDC modules (n = 224) (3.1%) (17.0%) (49.1%) (30.8%) 

Opportunity to extend learning when needed or 5 a9 104 76 
desired (n = 224) (2.2%) (17.4%) (46.4%) (33.9%) 


T12a. Overall, were you able to get the feedback and support you needed from your LDC 
coach (through written feedback in LDC CoreTools, or coaching and modeling in your 


LDC PLCs) to plan, teach, reflect on, and revise LDC modules)? 


(n = 224) 


Yes 216 teachers (96.4%) 
No 8 teachers (3.6%) 


T12b. Did your LDC coach provide written feedback on your module(s) in LDC CoreTools in a 


timely manner? 


(n = 224) 


Yes 211 teachers (94.2%) 
No 13 teachers (5.8%) 
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T13. Outside of the PLC meetings with your LDC coach, please indicate whether you used 
each of the following types of coach support, and how helpful you found these types 


of support. 
Used 
Did not 
use Not A little Moderately Very 
helpful helpful helpful helpful 

Written feedback in LDC CoreTools from 
your LDC coach (in the comments 31 6 a3 69 85 
areas and/or via the teacher work rubric) (13.8%) (2.7%) (14.7%) (30.8%) (37.9%) 
(n = 224) 
One-on-one Zoom video conference and/or 45 8 25 43 103 
call with your LDC coach (n = 224) (20.1%) (3.6%) (11.2%) (19.2%) (46.0%) 
Email or phone communication with your 58 4 3d. 60 ral 
LDC coach (n = 224) (25.9%) (1.8%) (13.8%) (26.8%) (31.7%) 
Other (n = 224) 162 3 5 18 36 
Please specify: (72.3%) (1.3%) (2.2%) (8.0%) (16.1%) 


T14. 


T15. 


Module Creation 


143 


n=192, Mean = 1.83, Range: 0-11 


n= 192, Mean = 0.91, Range: 0-5 


During the current school year (2018-19), how many LDC modules did you individually 
or collaboratively adapt from existing modules (e.g., modules you created in a prior 
year and/or modules found in the LDC Library in CoreTools)? 


| | Adapted modules 


During the current school year (2018-19), how many LDC modules did you create, 
either individually or with colleague(s)? Only include modules built from scratch, not 
those adapted from existing modules in the LDC library. 


| | New modules 


T16. How did members of your PLC collaborate to create LDC modules? 
Check all that apply. 
(n = 231) 
75 teachers (32.5%) 


Modules were created by individual teachers. 


Modules were created by teams of two or more teachers. 143 teachers (61.9%) 


Modules were created by the PLC as a whole. 49 teachers (21.2%) 


Other (please specify) 21 teachers (9.1%) 


T17. Please indicate to what extent you were able to do each of the following when 


creating LDC modules. 


Nott alain ccintebns fea cers lp a aeeee: 
extent extent 
Select focus standards for a writing 3 12 71 138 
assignment (n = 224) (1.3%) (5.4%) (31.7%) (61.6%) 
Create a standards-driven writing 4 14 66 140 
assignment (n = 224) (1.8%) (6.3%) (29.5%) (62.5%) 
Select high quality, complex texts and other 3 JA 99 98 
materials to engage students in deeper F P : . 
learning (n= 224} (1.3%) (10.7%) (44.2%) (43.8%) 
Identify the skills students need to develop 3 43 82 196 
to complete a writing assignment (n =224) (1.3%) (5.8%) (36.6%) (56.3%) 
Create daily lessons to teach the skills a 6 32 95 91 
student needs to complete a writing 2.79% 14.3% A2.A% 0.6% 
assignment (n = 224) (2.7%) (14.3%) (42.4%) (40.6%) 
Differentiate instruction by incorporating 
multiple ways of thinking, various levels of Z 40 100 77 
complexity, and multiple modalities. (3.1%) (17.9%) (44.6%) (34.4%) 
(n = 224) 
Plan for a variety of methods to assess a 7 37 98 32 
student progress (e.g., rubrics and/or mini- 3.1% 16.5% 3.8% 36.6% 
task scoring guides) (n = 224) (3.1%) (16.5%) (43.8%) (36.6%) 
Assess the quality of writing assignments 
and/or instructional plans using Peer 6 2 95 101 
Review/Curriculum Alignment Rubric (e.g. ‘ ‘ : 4 
Task Pitfalls Checklist, rubric indicators) (2.7%) (9.8%) (42.4%) (45.1%) 
(n = 224) 
Make a writing assignment relevant and 
engaging for students (n = 224) a is oe t12 
(1.3%) (7.1%) (41.5%) (50.0%) 
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Classroom Implementation 


T18. How many total LDC modules did you teach during the current school year (2018-19)? 


| | Modules n = 224, Mean = 2.16, Range: 0-6 


T19. Outside of modules, approximately how many individual LDC mini-tasks did you teach 
during the current school year (2018-19)? 


| | Mini-tasks n= 224, Mean = 3.33, Range: 0-20 


T20. Please indicate to what extent you were able to do each of the following activities 
when teaching LDC modules. 


Not at all’ |eeadneeienl tees eee 
extent extent 
Engage students in understanding the 2 15 89 118 
assignment and its rubric (n = 224) (0.9%) (6.7%) (39.7%) (52.7%) 
Engage students in accessing complex text 3 18 95 108 
for the purpose of the assignment (n =224) (1.3%) (8.0%) (42.4%) (48.2%) 
Systematically collect information about 5 23 T17 79 
students’ progress (n = 224) (2.2%) (10.3%) (52.2%) (35.3%) 
Provide feedback to students using 4 3]. 87 102 
assignment rubrics (n = 224) (1.8%) (13.8%) (38.8%) (45.5%) 
Locate evidence of standards in final student 2 22 85 115 
work on the writing assignment (n = 224) (0.9%) (9.8%) (37.9%) (51.3%) 
uc eee) ao " ce _ 
er ee (3.1%) (10.7%) (42.0%) (44.2%) 
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T21. Toward the beginning of the school year, did you “find and teach” a module from CoreTools? 


(n = 224) 
Yes 144 teachers (64.3%) 
No J sip to T24 80 teachers (35.7%) 


T22. What was the name of the Find and Teach module? 


T23. Did you make any adjustments to the Find and Teach module? 


(n = 144) 
Yes 104 teachers (72.2%) 
No 40 teachers (27.8%) 


T24. What module did you adapt, refine, and/or develop most during the current school year 
(2018-19)? This module is typically one you worked on after the Find and Teach module. 


T25. Which of these statements best describes how you created the module named in the 
previous question? 
(n = 224) 


| created a module from a template in CoreTools. 69 teachers (30.8%) 


| found and adjusted another teacher’s module from 155 teachers (69.2%) 
the LDC Library in CoreTools. 


T26. Did you teach this module in your classroom? 
(n = 224) 
Yes, | have already taught this module this year. 194 teachers (86.6%) 


No, but | plan to teach this module before the end of the 2017-18 school year. 16 teachers (7.1%) 


No, but | plan to teach this module during next school year. 9 teachers (4.0%) 


No. | do not currently have plans to teach this module in my classroom. 5 teachers (2.2%) 


Module Peer Review 


T27. Did you attend a Peer Review/Curriculum Alignment Workshop this school year? (Y/N) 


(n = 224) 
Yes 77 teachers (34.4%) 
No 147 teachers (65.6%) 
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T28. How many modules did you submit online for LDC National Peer Review during the 
current school year (2018-19)? 


| | Modules |f none, FJ siipto T30 n = 224, Mean = 0.58, Range: 0-3 


T29. How helpful did you find the National Peer Review process in improving the quality of 
your module? 


(n = 83) 
Not helpful 7 teachers (8.4%) 
A little helpful 19 teachers (22.9%) 
Moderately helpful 38 teachers (45.8%) 
Very helpful 19 teachers (22.9%) 
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| Impact on Teacher Practice and Learning 


T30. Between the beginning and end of this year’s work with LDC, please indicate how 
much your skills have improved in the following areas: 


Not at all A little Moderately | A great deal 
Selecting focus standards for a writing assignment 3 25 99 97 
(n = 224) (1.3%) (11.2%) (44.2%) (43.3%) 
Creating standards-driven writing assignments 3 30 95 96 
(n =2 24) (1.3%) (13.4%) (42.4%) (42.9%) 
Identifying the skills students need to develop to 3 27 102 92 
complete a writing assignment (n = 224) (1.3%) (12.1%) (45.5%) (41.1%) 
Creating daily lessons to teach the skills students need 5 32 109 78 
to complete a writing assignment (n = 224) (2.2%) (14.3%) (48.7%) (34.8%) 
Systematically collecting information on students’ 7 32 114 71 
progress. (n = 224) (3.1%) (14.3%) (50.9%) (31.7%) 
Identifying patterns of student understandings or vi 40 98 vie] 
misconceptions (n = 224) (3.1%) (17.9%) (43.8%) (35.3%) 
Using evidence of student progress on standards to 5 28 iy? 79 
modify subsequent instruction (n = 224) (2.2%) (12.5%) (50.0%) (35.3%) 
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T31. Please indicate the degree to which you agree or disagree with the statements below. 


auonsly Disagree Agree Stronely, 
disagree agree 
Participating in LDC raised my expectations for 3 Zt 117 83 
students’ writing. (n = 224) (1.3%) (9.4%) (52.2%) (37.1%) 
Using LDC modules became an important part of my 4 $1 115 54 
instructional practice. (n = 224) (1.8%) (22.8%) (51.3%) (24.1%) 
alert cesar ener te Niet ail ; si is = 
instruction, (n= 224) (1.8%) (15.2%) (58.0%) (25.0%) 
LDC helped me incorporate writing assignments into Z 25 118 79 
my existing curriculum. (n = 224) (0.9%) (11.2%) (52.7%) (35.3%) 
ie ae | ge * sl = 
fo} Lo} Loy fo} 
Professional Learning Community. (n = 224) (2.7%) (10.7%) (61.2%) (25.4%) 
LDC helped me improve on my teacher evaluation 8 48 118 50 
ratings. (n = 224) (3.6%) (21.4%) (52.7%) (22.3%) 
Participating in LDC helped me develop working 7 40 111 66 
relationships with teachers in different grades and/or 3.1% 17.9% 9.6% 29.59% 
subjects. (n = 224) (3.1%) (17.9%) (49.6%) (29.5%) 
| shared my LDC work with colleagues outside of the 14 of 88 45 
LDC PLC. (n = 224) (6.3%) (34.4%) (39.3%) (20.1%) 
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Impact on Student Learning 


T32. Please indicate to what extent LDC had a positive effect on students in the following 


areas. 
Not at all A little Moderately | A great deal 
vi 41 143 63 
Reading skills (n = 224 
a akc (3.1%) (18.3%) (50.4%) (28.1%) 
Content knowledge (n = 224) 2 pi na ia 
(0.9%) (15.2%) (43.8%) (40.2%) 
7 2 ; 4 30 101 89 
Ability to complete writing assignments (n = 224) (1.8%) (13.4%) (45.1%) (39.7%) 
3 32 103 86 
lity of students’ writi = 224 
BUalyP StS aenS A Ee! (1.3%) (14.3%) (46.0%) (38.4%) 
7 46 98 73 
Coll d dy skills (n = 224 
a ac (3.1%) (20.5%) (43.8%) (32.6%) 
Capacity to analyze and understand the 2 37 a5 86 
components of a writing assignment (n = 224) (0.9%) (16.5%) (44.2%) (38.4%) 
5 49 105 65 
Speaki d listening skills (n = 224 
PeSsine gue isrenity Silene) (2.2%) (21.9%) (46.9%) (29.0%) 
Overall literacy performance (n = 224) : ae a6 in 
a 7 (1.3%) (14.7%) (51.8%) (32.1%) 
Performance on assessments throughout the 5 50 101 68 
school year (n = 224) (2.2%) (22.3%) (45.1%) (30.4%) 
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Teacher Leader Support 


The following question refers to the LDC project liaison in your school. This is the teacher leading your 


Professional Learning Community work. 


T33. Please indicate the degree to which you agree or disagree with the following 


statements. 

Strongly Dieaeree Reree Strongly 

disagree agree 
Our school’s LDC teacher leader effectively 0 5 63 124 
supported our Professional Learning Community 0.0% 2.6% 32.3% 64.6% 
meetings. (n = 192) (0.0%) (2.6%) (32.8%) (64.6%) 
When | had questions about LDC, | felt comfortable 0 6 58 128 
approaching our school’s teacher leader. (n = 192) (0.0%) (3.1%) (30.2%) (66.7%) 
Our teacher leader helped teachers align LDC to 0 13 67 112 
broader school instructional goals. (n = 192) (0.0%) (6.8%) (34.9%) (58.3%) 
Our teacher leader offered useful feedback for the 0 ca 69 112 
design and revision of LDC modules. (n = 192) (0.0%) (5.7%) (35.9%) (58.3%) 
Our teacher leader was effective in inviting 0 2 75 115 
teachers to join the LDC initiative. (n = 192) (0.0%) (1.0%) (39.1%) (59.9%) 


Ke) School Administrator Support 


The following questions refer to the school administrator who oversees the LDC project at your school. 


T34. What proportion of PLC meetings focused on LDC did your school administrator attend? 


Less than one quarter of LDC PLCs 


About one quarter of LDC PLCs 
About one half of LDC PLCs 


About three quarters of LDC PLCs 


More than three quarters of LDC PLCs 
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(n =224) 


45 teachers (20.1%) 
26 teachers (11.6%) 
50 teachers (22.3%) 
32 teachers (14.3%) 
71 teachers (31.7%) 


T35. How many times did your school administrator observe you teach an LDC mini-task 
during the current school year (2018-19)? 


(n = 224) 
0 times 85 teachers (37.9%) 
1 time 50 teachers (22.3%) 
2 times 55 teachers (24.6%) 
3 or more times 34 teachers (15.2%) 


T36. Please indicate the degree to which you agree or disagree with the following 


statements. 
My school administrator... strongly Disagree Agree etrongly 
disagree agree 

. . _ Ae. 27 126 59 
had a firm understanding of LDC. (n = 224) (5.4%) (12.1%) (56.3%) (26.3%) 
allocated resources such as teacher time, 
payment, administrator time, support 8 23 115 78 
staff, sub coverage, etc., to ensure the LDC (3.6%) (10.3%) (51.3%) (34.8%) 
team could meet. (n = 224) 
encouraged teachers to participate in LDC. Z 15 113 94 
(n = 224) (0.9%) (6.7%) (50.4%) (42.0%) 
expressed concerns that implementing LDC 60 79 55 30 
is taking time away from other 26.8% 35.3% 24.6% 13.4% 
instructional priorities. (n = 224) (26.8%) (35.3%) (24.6%) (13.4%) 
ce ened) oe 29 ik 57 
ae = oa (4.5%) (12.9%) (57.1%) (25.4%) 
provided me with feedback about my LDC 25 57 103 39 
planning and/or instruction. (n =224) (11.2%) (25.4%) (46.0%) (17.4%) 
made formative assessment a priority at 13 39 121 51 
my school. (n = 224) (5.8%) (17.4%) (54.0%) (22.8%) 
used LDC to implement standards-driven 15 32 120 57 
assignments within existing curriculum. 4 é é é 
(n= 224) (6.7%) (14.3%) (53.6%) (25.4%) 
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iti) Teacher Leadership Role 


T37. Please indicate the degree to which you agree or disagree with the following 
statements about your role in your school’s LDC implementation. 


SrCuely Disagree Agree suiengly 
disagree agree 
| was involved in setting instructional goals for the LDC 10 48 95 39 
work at my school. (n = 192) (5.2%) (25.0%) (49.5%) (20.3%) 
| was involved in discussions about how to expand LDC A 70 V7 34 
implementation at my school in future years. (n = 192) (5.7%) (36.5%) (40.1%) (17.7%) 
epee eb ere, | ae - a . 
eee coe "| (6.8%) | (28.6%) | (43.2%) | (21.4%) 
am interested in learning more about how to lead ie 1 72 72 7 
implementation at my school by facilitating with the virtual 10.9% 37.5% 37.5% 14.1% 
coach, providing feedback to my peers, etc. (n = 192) (10.9%) (37.5%) (37.5%) (14.1%) 
sy Facilitators and Barriers 
T38. Please indicate the degree to which you agree or disagree with the following 
statements. 
strongly Disagree Agree stenely 
disagree agree 
My LDC PLC was given sufficient time to meet. 35 128 52 
(n = 224) (4.0%) (15.6%) (57.1%) (23.2%) 
| felt adequately prepared to effectively implement 26 140 50 
LDC modules in my classroom. (n = 224) (3.6%) (11.6%) (62.5%) (22.3%) 
It was challenging to find content-rich reading 14 83 104 23 
materials for the LDC modules | developed. (n = 224) (6.3%) (37.1%) (46.4%) (10.3%) 
My school had adequate technology to support 16 115 90 
teachers’ use of LDC. (n = 224) (1.3%) (7.1%) (51.3%) (40.2%) 
It was easy to find and adapt LDC mini-tasks for use in 35 135 47 
my classroom. (n = 224) (3.1%) (15.6%) (60.3%) (21.0%) 
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13 Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 
e CoreTools online platform 
e LDC online courses in the “Learn” section of CoreTools 
e  =Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e =|In-person coaching 


o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


T39. What supports did you find the most useful and why? 


T40. What supports were not helpful and why? 


T41. In what ways could LDC implementation be improved in your school in the future? 
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Appendix B: 
Teacher Leader Survey and Responses 


2018-2019 
LDC Teacher Leader Survey 


i) LDC Participation 


TL1. Prior to the current school year (2018-19), did you have any experience with LDC? 


(n = 37) 
Yes 34 teacher leaders (91.9%) 
No y Skip toTL3 3 teacher leaders (8.1%) 


TL2. How many of the following did you teach prior to the current school year (2018-19)? 


| | LDC modules n = 34, Mean = 2.38, Range: 0-10 


| | LDC mini-tasks, outside of modules n= 34, Mean = 3.06, Range: 0-16 


Professional Learning Community and Teacher Collaboration 


The following questions involve the LDC Professional Learning Community (PLC) that you are leading. 


TL3. About how often did your LDC PLC meet? 


(n=37) 
Less than once a month 1 teacher leader (2.7%) 
Once a month 8 teacher leaders (21.6%) 
Every other week 24 teacher leaders (64.9%) 
Once a week ib Skip to TL5 3 teacher leaders (8.1%) 
Twice a week or more often y Skip to TLS 1 teacher leader (2.7%) 
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TL4. What were the primary barriers preventing your LDC PLC from meeting weekly? 
Select all that apply. 


(n= 33) 

7 teacher leaders (21.2%) 
9 teacher leaders (27.3%) 
4 teacher leaders (12.1%) 


PLC time was not protected. 


PLC members had limited interest in attending meetings. 


School administrator did not make it a priority. 


| was unable to provide sufficient organizational support. 1 teacher leader (3.0%) 


3 teacher leaders (9.1%) 


Not enough teachers participated. 


PLC members had other priorities that competed with LDC 15 teacher leaders (45.5%) 


participation. 


Other (please specify) 


0 teacher leaders (0.0%) 


TL5. About how often did you have informal discussions (as opposed to scheduled 
meetings) about LDC with teachers in your LDC PLC? 


Less than once a month 


Once a month 


Every other week 


Once a week 


Twice a week or more 


(n = 37) 

2 teacher leaders (5.4%) 
11 teacher leaders (29.7%) 
10 teacher leaders (27.0%) 
10 teacher leaders (27.0%) 
4 teacher leaders (10.8%) 


TL6. On average how long did your school’s LDC PLC meetings typically last? 


Less than 45 minutes 


45 to 59 minutes 


60 to 74 minutes 


75 minutes or more 


(n = 37) 

1 teacher leader (2.7%) 

22 teacher leaders (59.5%) 
12 teacher leaders (32.4%) 
2 teacher leaders (5.4%) 
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Teacher Training and Support 


TL7. How effective was your LDC PLC in the following areas? 


Not A little Moderately Very 
effective effective effective effective 
cing oemene menartenare | ovoom | acoee) | 1407.8% | 19(614%) 
joerg eurormentters ences stare | ofoow) | 4 (tox) | 14(37.8%) | 19 (61.4% 
Allowing space to share student work (n = 37) 0 (0.0%) 5 (13.5%) 15 (40.5%) | 17 (45.9%) 
Helping teachers learn to improve their LDC 0 (0.0%) 4 (10.8%) 15 (40.5%) | 18 (48.6%) 


instructional plans. (n = 37) 


TL8. How would you rate each of the following aspects of the online course material (in the 
Learn tab in LDC CoreTools) that your coach used or directed you to use? 


Poor Fair Good Excellent 
Clarity of information presented (n = 37) 0 (0.0%) 5 (13.5%) 16 (43.2%) | 16 (43.2%) 
Relevance of information presented (n = 37) 0 (0.0%) 4 (10.8%) 12 (32.4%) | 21 (56.8%) 
Ease of use (n = 37) 3 (8.1%) 6 (16.2%) 18 (48.6%) | 10 (27.0%) 
Usefulness of resource documents (e.g., LDC 
Pitfall Checklist, CCSS Mental Markers, etc.) 1 (2.7%) 5 (13.5%) 13 (35.1%) | 18 (48.6%) 
(n = 37) 
Usefulness of videos (n = 37) 3 (8.1%) 7 (18.9%) 18 (48.6%) 9 (24.3%) 
Degree to which course material helped teachers n _ . : 
to create and/or adapt LDC modules (n = 37) ki, PSO) ATV) | TESS Aes 
Opportunity to extend learning when needed 0 (0.0%) 5 (13.5%) 16 (43.2%) 16 (43.2%) 


or desired (n = 37) 


157 


TL9a. Overall, were you able to get the feedback and support you needed from your LDC 
coach (through written feedback in LDC CoreTools, or coaching and modeling in your 
LDC PLCs) to plan, teach, reflect on, and revise LDC modules? 


(n = 37) 


Yes 36 teacher leaders (97.3%) 
No 1 teacher leader (2.7%) 


TL9b. Did your LDC coach provide written feedback on your module(s) in LDC CoreTools in a 


timely manner? 


(n = 37) 


Yes 36 teacher leaders (97.3%) 
No 1 teacher leader (2.7%) 


TL10. Outside of the PLC meetings with your LDC coach, please indicate whether you used 
each of the following types of coach support, and how helpful you found these types 


of support. 
Used 
Did not use Not A little Moderately Very 
helpful helpful helpful helpful 

Written feedback in LDC CoreTools 
from your LDC coach (in the 

: 3 (8.1%) 0 (0.0%) 4 (10.8%) 7 (18.9%) 23 (62.2%) 
comments areas and/or via 
the teacher work rubric) (n = 37) 
One-on-one Zoom video conference 
and/or call with your LDC coach 4 (10.8%) 0 (0.0%) 5 (13.5%) 2 (5.4%) 26 (70.3%) 
(n = 37) 
Email or phone communication with " J és " ‘ 
Your LDC coach 287) 1 (2.7%) 1 (2.7%) 4 (10.8%) 7 (18.9%) 24 (64.9%) 
Other (please specify) (n = 37) 27 (73.0%) 1 (2.7%) 1 (2.7%) 1 (2.7%) 7 (18.9%) 
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TL11. How many in-person and/or online LDC professional development offerings for school 
administrators and teacher leaders did you attend during the current school (e.g., Summer 
in-person launch days, Quarterly in-person teacher leader meetings, LDC monthly virtual 
coach meetings)? 


| | Professional development offerings n= 37, Mean = 6.00, Range: 0-20 


Support to Teacher Leaders from LDC Coach 


TL12. Please indicate the degree to which you agree or disagree with the following 


statements. 


Strengly Disagree Agree strengly N/A 

disagree agree 
| was able to reach my LDC coach if | ‘, é z J ‘ 
had'any questions about (DC. (Masa) 0 (0.0%) 2 (5.4%) 6 (16.2%) | 28 (75.7%) 1 (2.7%) 
LDC provided adequate technical 
support for issues with the CoreTools 0 (0.0%) 1 (2.7%) 8 (21.6%) | 22 (59.5%) | 6 (16.2%) 
online platform. (n = 37) 
LDC offered sufficient professional 
development opportunities for me to ‘ e ' é 2 
iead the initiative in my-school, 0 (0.0%) 0 (0.0%) 11 (29.7%) | 24 (64.9%) 2 (5.4%) 
(n = 37) 
LDC coaches were able to connect me 
with additional resources when 0 (0.0%) 2 (5.4%) 8 (21.6%) | 25 (67.6%) 2 (5.4%) 
needed. (n = 37) 
It was challenging to coordinate with 
our LDC coach on/how to structure =| > (a2.4m) | 912430) | e(ae2%) | e(2Lex) | 215.4%) 
Professional Learning Community 
time. (n = 37) 
When | reached out to our LDC coach, 4 i Pe és J 
he orshe responded quickly. (n=a7) 0 (0.0%) 2 (5.4%) 4 (10.8%) | 30 (81.1%) 1 (2.7%) 
ve = coach was easy toworkwith | 4 go) | 4 (2.7%) | 3(8.1%) | 32 (86.5%) | 1(2.7%) 
Our LDC coach was knowledgeable 
and provided high quality guidance. 0 (0.0%) 0 (0.0%) 4 (10.8%) | 32 (86.5%) 1 (2.7%) 
(n = 37) 
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Module Creation 


TL13. During the current school year (2018-19), how many LDC modules did your PLC 
individually or collaboratively adapt from existing modules (e.g., modules created in 
a prior year and/or modules from the LDC Library in CoreTools)? 


| | Adapted modules n = 37, Mean = 2.92, Range: 0-12 


TL14. During the current school year (2018-19), how many LDC modules did your PLC create 
(either individually or in a group)? Only include modules built from scratch, not those 
adapted from existing modules in the LDC library. 


| | New modules n = 37, Mean = 1.38, Range: 0-6 


TL15. How did members of your PLC collaborate to create LDC modules? Check all that 


apply. 
(n = 37) 
Modules were created by individual teachers. 17 teacher leaders (45.9%) 
Modules were created by teams of two or more teachers. 22 teacher leaders (59.5%) 
Modules were created by the PLC as a whole. 10 teacher leaders (27.0%) 
Other (please specify) 4 teacher leaders (10.8%) 
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TL16. Please indicate to what extent you were able to do each of the following when 


creating LDC modules. 


ete To a great 
Not at all A little bit moderate pie 
extent 

Select focus standards for a writing é . z e 
assianmentwim=a7) 0 (0.0%) 1(2.:7%) 7 (18.9%) 29 (78.4%) 
Create a standards-driven writing ‘ . i . 
dssignmentiueail 0 (0.0%) 1 (2.7%) 8 (21.6%) 28 (75.7%) 
Select high quality, complex texts and other 
materials to engage students in deeper 0 (0.0%) 2 (5.4%) 13 (35.1%) | 22 (59.5%) 
learning (n = 37) 
Identify the skills students need to develop 
to complete a writing assignment (n = 37) 0 (0.0%) 2 (5.4%) 8 (21.6%) 27 (73.0%) 
Create daily lessons to teach the skills a 
student needs to complete a writing 0 (0.0%) 4 (10.8%) 16 (43.2%) | 17 (45.9%) 
assignment (n = 37) 
Differentiate instruction by incorporating 
multiple ways of thinking, various levels of 1 (2.7%) 5 (13.5%) 14 (37.8%) 17 (45.9%) 
complexity, and multiple modalities. 
(n = 37) 
Plan for a variety of methods to assess 
student progress (e.g., rubrics and/or mini- 2 (5.4%) 4 (10.8%) 17 (45.9%) | 14 (37.8%) 
task scoring guides) (n = 37) 
Assess the quality of writing assignments 
and/or instructional plans using Peer 
Review/Curriculum Alignment Rubric (e.g. 1 (2.7%) 4 (10.8%) 11 (29.7%) | 21 (56.8%) 
Task Pitfalls Checklist, rubric indicators) 
(n =37) 
Make a writing assignment relevant and 0 (0.0%) 2 (5.4%) 10 (27.0%) | 25 (67.6%) 


engaging for students (n = 37) 
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Impact on Student Learning 


TL17.Please indicate to what extent LDC had a positive effect on students in the following areas. 


Not at all A little Moderately | A great deal 
Reading skills (n = 37) 1 (2.7%) 5 (13.5%) 14 (37.8%) | 17 (45.9%) 
Content knowledge (n = 37) 0 (0.0%) 5 (13.5%) 12 (32.4%) | 20 (54.1%) 
Ability to complete writing assignments (n = 37) 0 (0.0%) 6 (16.2%) 10 (27.0%) | 21 (56.8%) 
Quality of students’ writing (n = 37) 0 (0.0%) 6 (16.2%) 9 (24.3%) 22 (59.5%) 
College and career ready skills (n = 37) 0 (0.0%) 10 (27.0%) | 10(27.0%) | 17 (45.9%) 
lait ee re the components | 9 (0.0%) | 7 (18.9%) | 8 (21.6%) | 22 (59.5%) 
Speaking and listening skills (n = 37) 0 (0.0%) 7 (18.9%) 13 (35.1%) | 17 (45.9%) 
Overall literacy performance (n = 37) 0 (0.0%) 5 (13.5%) 13 (35.1%) | 19 (51.4%) 
Sees on assessments throughout the school 1 (2.7%) 6 (16.2%) 13 (35.1%) | 17 (45.9%) 
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School Administrator Support 


The following questions refer to the school administrator who oversees the LDC project at your school. 


TL18. What proportion of PLC meetings focused on LDC did your school administrator 


attend? 


Less than one quarter of LDC PLCs 


About one quarter of LDC PLCs 
About one half of LDC PLCs 
About three quarters of LDC PLCs 


More than three quarters of LDC PLCs 
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(n = 37) 

9 teacher leaders (24.3) 
4 teacher leaders (10.8%) 
9 teacher leaders (24.3%) 
6 teacher leaders (16.2%) 
9 teacher leaders (24.3%) 


TL19. Please indicate the degree to which you agree or disagree with the following 


statements. 
My school administrator... Sirenely Disagree Agree Strongly 
disagree agree 
had a firm understanding of LDC. (n = 37) 2 (5.4%) 6 (16.2%) | 21 (56.8%) 8 (21.6%) 


allocated resources such as teacher time, payment, 


administrator time, support staff, sub coverage, etc., 1 (2.7%) 5 (13.5%) 17 (45.9%) 14 (37.8%) 
to ensure the LDC team could meet. (n = 37) 


encouraged teachers to participate in LDC. (n = 37) 0 (0.0%) 3 (8.1%) 15 (40.5%) 19 (51.4%) 


expressed concerns that implementing LDC is taking 


fo} fo} fo} Lo} 
time away from other instructional priorities. (n = 37) 1535179) | 25 (30.5%) 616.2%) 3 (8.1%) 


communicated how using LDC’s tools supported 


Loy fo} fo} Loy 
specific school initiatives and/or goals. (n = 37) Ae ea) 7 (18.9%) 17 (45.9%) seve) 


provided me with feedback about my LDC planning 


0, 0, 0, 0, 
ra/orinsiucton lames 1(2.7%) | 11(29.7%) | 20(54.1%) | 5 (13.5%) 


made formative assessment a priority at my school. 


(n = 37) 0 (0.0%) 7 (18.9%) | 23 (62.2%) 7 (18.9%) 


Used LDC to implement standards-driven assignments 


iv) i) i) iv) 
within existing curriculum. (n = 37) 1 (2.7%) 8 (24.6%) 16 (43.27%) a dtd 
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Teacher Leader Leadership Role 


TL20. Please indicate the degree to which you agree or disagree with the following 
statements about your role in leading your school’s LDC implementation. 


I I 
Strong UJ Disagree Agree strongly 
disagree agree 


| met regularly with my school administrator to make 


Le} iv) iv) 9 
planning decisions around LDC. (n = 37) O(0.076) SSE) | 47 (95-97) Tee) 


| was involved in discussions about differentiating LDC 
implementation to meet teacher learning needs. 0 (0.0%) 4 (10.8%) 19 (51.4%) | 14 (37.8%) 
(n = 37) 


| was involved in discussions about how to expand LDC 


fo} 9 fo} fo} 
implementation at my school in future years. (n = 37) 0 (0.0%) 7 (18.9%) 17 (45.9%) | 13 (35.1%) 


My role as an LDC teacher leader allowed me to 
effectively advocate for additional resources on my 0 (0.0%) 8 (21.6%) | 19(51.4%) | 10 (27.0%) 
campus. (n = 37) 


| was involved in adjusting the problems of practice 


0, fo} fo} Lo} 
that my school targeted with the LDC work. (n = 37) 0 (0.0%) 6 (16.2%) i el a a 


| met regularly with my LDC coach to manage the LDC 


Gone 0 (0.0%) | 6(16.2%) | 9 (24.3%) | 22 (59.5%) 


| feel that my position as an LDC lead teachers allowed 
me to build my capacity as an instructional leader 0 (0.0%) 3 (8.1%) 18 (48.6%) | 16 (43.2%) 
among my colleagues. (n = 37) 


lam confident that | can lead our LDC PLC in the 
future without the assistance of an LDC coach. 1 (2.7%) 5 (13.5%) 18 (48.6%) | 13 (35.1%) 
(n = 37) 
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Alignment 


TL21. Please indicate the degree to which you agree or disagree with the following 


statements. 
strongly Disagree Agree strongly 
disagree agree 
Our school connected LDC implementation to our 
specie schuslwide eoale it Lay) 2 (5.4%) 10 (27.0%) | 11 (29.7%) 14 (37.8%) 
LDC helped teachers create writing assignments to : . Z s 
use within their current curricula. (n = 37) 0 (0.0%) (L087) 22 (39.570) i i 
eae other initiatives taking place in 0 (0.0%) 8 (21.6%) 19 (51.4%) 10 (27.0%) 
| view LDC as a strategy for implementing my state’s = : . a 
College- and Career-Ready Standards. (n = 37) 0 (0.0%) 3 (8.1%) 21,(36.829) 13,(31,5%) 
The time spent implementing LDC interfered with . é e . 
other important initiatives at my school. (n = 37) Oe 2e) | Sasa 14 (37.8%) 4 (10.876) 
LDC helped prepare students in my school for 
current state assessments. (n = 37) 0 (0.0%) 2 (ae 22. {59.5%} 13 (35.1%) 
It was difficult for teachers to focus on LDC because ‘ ‘ . " 
of other competing priorities at the school. (n = 37) 2 (3.4%) 7 (18.9%) 22 (59.5%) 6 (16.2%) 
Our instructional leaders are using LDC to implement 
standards-driven assignments within the existing 2 (5.4%) 7 (18.9%) 20 (54.1%) 8 (21.6%) 


curriculum. (n = 37) 


165 


180) Scale-up and Sustainability 


TL22. Please indicate the degree to which you agree or disagree with the following 


statements. 

strongly Disagree Agree strongly 
disagree agree 

| expect that most teachers participating in 

LDC this year will continue to do so next 0 (0.0%) 14 (37.8%) 16 (43.2%) 7 (18.9%) 

year. (n = 37) 

Teachers at my school who were not part of 

the LDC PLC meetings used the LDC 8 (21.6%) 15 (40.5%) 12 (32.4%) 2 (5.4%) 


planning process and/or LDC CoreTools. 
(n = 37) 


As a result of LDC, new collaborations 
across grades and/or subjects were created 3 (8.1%) 14 (37.8%) 15 (40.5%) 5 (13.5%) 
or are being launched at my school. (n = 37) 


Teachers and administrators at my school 
are committed to sustaining the LDC 1 (2.7%) 13 (35.1%) 17 (45.9%) 6 (16.2%) 
initiative. (n = 37) 


| expect our LDC PLC to increase in size next 


negra 3(8.1%) | 22(59.5%) | 9 (24.3%) 3 (8.1%) 


iM) District Support 


TL23. Please indicate the degree to which you agree or disagree with the following 
statements. 


Strongl Strongl Don’t know 
A aM Disagree Agree Bly 
disagree agree 


District leaders supported the 


mplentéuistion st ibe maa 0 (0.0%) | 4(10.8%) | 14 (37.8%) | 11(29.7%) | 8 (21.6%) 


District leaders had a firm understanding of 


ioeda 89) 0 (2.1%) | 7(18.9%) | 7 (18.9%) | 10(27.0%) | 13 (35.1%) 


District leaders are interested in spreading 


the use of LDC to additional schools. (n = 37) 0 (0.0%) 6 (16.2%) 5 (13.5%) LO(2 7.0%) | 16 (43.2%) 


District professional development efforts 


iv) i) i) i) i) 
were aligned with the LDC initiative. (n = 37) ate) TU8.2%8) mine -2r) PER) | See) 


District leaders visited my school to discuss 


0, 0, 0 0, 0, 
sha implementainotinc IMeeml 5 (13.5%) | 9 (24.3%) | 8 (21.6%) | 9(24.3%) | 6 (16.2%) 
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12 Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 
e CoreTools online platform 
e LDC online courses in the “Learn” section of CoreTools 
e =Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e =In-person coaching 


o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


TL24. What supports did you find the most useful and why? 


TL25. What supports were not helpful and why? 


TL26. In what ways could LDC implementation be improved in your school in the future? 
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Appendix C: 


Administrator Survey and Responses 


2018-2019 
LDC School Administrator Survey 


LDC Participation 


A1. What is your role at the school? 


(n = 35) 


Principal 


23 admins (65.7%) 


Assistant Principal 


8 admins (22.9%) 


Other (please specify) 


4 admins (11.4%) 


Professional Learning Community 


A2. What proportion of LDC Professional Learning Community (PLC) meetings did you 


attend during the current school year? 


(n = 35) 


Less than one quarter of LDC PLCs 


6 admins (17.1%) 


About one quarter of LDC PLCs 


7 admins (20.0%) 


About one half of LDC PLCs 


8 admins (22.9%) 


About three quarters of LDC PLCs 


7 admins (20.0%) 


More than three quarters of LDC PLCs 


7 admins (20.0%) 
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Training and Support 


A3. How many in-person and/or online LDC professional development offerings for school 
administrators and teacher leaders did you attend during the current school year (e.g., 
Summer launch meetings, quarterly in-person administrator meetings, monthly virtual 
LDC leadership meetings)? 


| | LDC Professional development offerings n= 35, Mean =5.09, Range: 1-20 


A4. Please indicate the degree to which you agree or disagree with the following statements. 


Strongly Disagree Agree strongly N/A 
disagree agree 
| was able to reach LDC staff when | had 0 (0.0%) 0 (0.0%) 6 (17.1%) 29 (82.9%) 0 (0.0%) 
questions about LDC. (n = 35) 
My school has adequate technology to 0 (0.0%) 0 (0.0%) 6 (17.1%) 29 (82.9%) 0 (0.0%) 
access LDC online resources. (n = 35) 
LDC offered sufficient professional 0 (0.0%) 0 (0.0%) 11 (31.4%) 24 (68.6%) 0 (0.0%) 


development opportunities for LDC 
teacher leaders. (n = 35) 


LDC offered sufficient professional 0 (0.0%) 0 (0.0%) 10 (28.6%) 25 (71.4%) 0 (0.0%) 
development opportunities for school 
administrators. (n = 35) 


LDC staff members were able to connect 0 (0.0%) 0 (0.0%) 7 (20.0%) 27 (77.1%) 1 (2.9%) 
me with additional resources when 
needed. (n = 35) 
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Classroom Observation 


A5. On average, how many times during the school year did you observe each member of 
the LDC PLC teaching an LDC module? 


(n= 35) 
0 times Qsiiv to A7 2 admins (5.7%) 
1 time 12 admins (34.3%) 
2 times 7 admins (20.0%) 
3 or more times 14 admins (40.0%) 


A6. On average, how effective were LDC modules in developing students’ literacy skills? 


(n = 33) 
Not effective 0 admins (0.0%) 
A little effective 2 admins (6.1%) 
Moderately effective 15 admins (45.5%) 
Very effective 16 admins (48.5%) 
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Impact on Teacher Practice 


A7. Based on your oversight of the LDC program, please indicate on average how much the 
teaching practice of LDC PLC members improved in each of the following areas: 


Not at all A little Moderately | A great deal 
Selecting a set of focus standards for a writing 0 (0.0%) 3 (8.6%) 12 (34.3%) 20 (57.1%) 
assignment (n = 35) 
Creating standards-driven writing assignments (n = 35) 0 (0.0%) 2 (5.7%) 17 (48.6%) 16 (45.7%) 
Identifying the skills students need to develop to 0 (0.0%) 3 (8.6%) 19 (54.3%) 13 (37.1%) 


complete a writing assignment (n = 35) 


Creating daily lessons to teach each skill a student needs 0 (0.0%) 6 (17.1%) 21 (60.0%) 8 (22.9%) 
to complete a writing assignment (n = 35) 


Systematically collecting information on students’ 0 (0.0%) 7 (20.0%) 20 (57.1%) 8 (22.9%) 
progress. (n = 35) 

Identifying patterns of student understandings or 0 (0.0%) 5 (14.3%) 19 (54.3%) 11 (31.4%) 
misconceptions (n = 35) 

Using evidence of student progress on standards to 0 (0.0%) 6 (17.1%) 21 (60.0%) 8 (22.9%) 


modify subsequent instruction (n = 35) 
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Impact on Student Learning 


A8. Please indicate to what extent LDC had a positive effect on students in the following 


areas: 


school year (n = 35) 


Not at all A little Moderately | A great deal 
Reading skills ae 38) 0 (0.0%) 10 (28.6%) 18 (51.4%) 7 (20.0%) 
Pais tauicdesnen 0 (0.0%) 6 (17.1%) 15 (42.9%) | 14 (40.0%) 

0 (0.09% 1299 18 (51.49 16 (45.79 
Ability to complete writing assignments (n = 35) ie) ae) ( %) ( #) 
Quality of students’ writing (n = 35) ce aid) eae sl cd 

0, 0, 9 9 

College and career ready skills (n = 35) a F12ONS) di cea 
Capacity to analyze and understand the 0 (0.0%) 4 (11.4%) 13 (37.1%) 18 (51.4%) 
components of a writing assignment (n = 35) 

UN299 Oi 25.79 16 (45.79 9 (25.77 
Speaking and listening skills (n = 35) ald) ( ~) ( - ( _) 

0 (0.09% 5 (14.39 23 (65.79 7 (20.09 
Overall literacy performance (n = 35) one) ( mn) ( “ ( .) 
Performance on assessments throughout the 0 (0.0%) 10 (28.6%) 20 (57.1%) 5 (14.3%) 
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7 


Administrator Leadership Role 


A9. Please indicate the degree to which you agree or disagree with the following 


statements about your role in leading LDC implementation in your school: 


I I 
strong : Disagree Agree sone ly 
disagree agree 

| was able to shape LDC implementation at my school. 1 (2.9%) 4 (11.4%) 23 (65.7%) 7 (20.0%) 
(n = 35) 

| met regularly with the LDC teacher leader in my school 0 (0.0%) 3 (8.6%) 27 (77.1%) 5 (14.3%) 

to stay abreast of implementation progress. (n = 35) 

| was involved in discussions about differentiating LDC 0 (0.0%) 3 (8.6%) 28 (80.0%) 4 (11.4%) 

implementation to meet teacher learning needs. (n = 35) 

| led discussions about how to expand my school’s LDC 1 (2.9%) 7 (20.0%) 22 (62.9%) 5 (14.3%) 

implementation in future years. (n = 35) 

| made changes to school schedules to accommodate LDC 1 (2.9%) 4 (11.4%) 14 (40.0%) 16 (45.7%) 

professional learning time. (n = 35) 

| allocated resources such as teacher time, payment, 1 (2.9%) 2 (5.7%) 16 (45.7%) 16 (45.7%) 


administrator time, support staff, sub coverage, etc., to 
ensure the LDC team could meet. (n = 35) 
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Alignment 


A10. Please indicate the degree to which you agree or disagree with the following 


statements: 

t I I 
strong Mu Disagree Agree strongly 
disagree agree 

Our school connected LDC implementation to our 0 (0.0%) 6 (17.1%) 21 (60.0%) 8 (22.9%) 
specific schoolwide goals. (n = 35) 

LDC helped teachers create writing assignments to 0 (0.0%) 2 (5.7%) 23 (65.7%) | 10 (28.6%) 
use within their current curricula. (n = 35) 

LDC complemented other initiatives taking place in 0 (0.0%) 1 (2.9%) 25 (71.4%) 9 (25.7%) 
my school. (n = 35) 

| view LDC as a strategy for implementing my state’s 0 (0.0%) 2 (5.7%) 23 (65.7%) | 10 (28.6%) 
College- and Career-Ready Standards. (n = 35) 

The time spent implementing LDC interfered with 8 (22.9%) 14 (40.0%) | 10 (28.6%) 3 (8.6%) 
other important initiatives at my school. (n = 35) 

LDC helped prepare students in my school for 0 (0.0%) 3 (8.6%) 25 (71.4%) 7 (20.0%) 
current state assessments. (n = 35) 

It was difficult for teachers to focus on LDC because 4 (11.4%) 16 (45.7%) | 11 (31.4%) 4 (11.4%) 
of other competing priorities at the school. (n = 35) 

lam using LDC to implement standards-driven 1 (2.9%) 3 (8.6%) 23 (65.7%) 8 (22.9%) 
assignments within the existing curriculum. (n = 35) 

LDC helped improve teacher evaluation ratings. 1 (2.9%) 11 (31.4%) | 20(57.1%) 3 (8.6%) 
(n = 35) 
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Scale-up and Sustainability 


A11. Please indicate the degree to which you agree or disagree with the following 


statements: 
Strongly Dieanree peice Strongly Don’t 
disagree agree know 
| expect that most teachers participating in LDC this 0 3 20 9 3 
year will continue to do so next year. (n = 35) (0.0%) (8.6%) (57.1%) (25.7%) (8.6%) 
Teachers at my school who were not part of the LDC 1 21. 7 3 2 
PLC meetings used the LDC planning process and/or (2.9%) (60.0%) (20.0%) (8.6%) (8.6%) 
LDC CoreTools. (n = 35) 
Asa result of LDC, new collaborations across grades 2 vi 19 5 2 
and/or subjects were created or are being launched (5.7%) (20.0%) (54.3%) (14.3%) (5.7%) 
at my school. (n = 35) 
Teachers and administrators at my school are 0 4 22 8 1 
committed to sustaining the LDC initiative. (n = 35) (0.0%) (11.4%) (62.9%) (22.9%) (2.9%) 
| expect our LDC PLC to increase in size 1 16 12 4 2 
next year. (n = 35) (2.9%) (45.7%) (34.3%) (11.4%) (5.7%) 
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18 @)) District Support 


A12. Please indicate the degree to which you agree or disagree with the following 


statements: 
I Don’ 
Strongly Disagree Acie Strongly on’t 
disagree agree know 
District leaders supported the 0 (0.0%) 5 (14.3%) 15 (42.9%) 6 (17.1%) 9 (25.7%) 
implementation of LDC. (n = 35) 
District leaders had a firm understanding 0 (0.0%) 8 (22.9%) | 13 (37.1%) | 4(11.4%) | 10 (28.6%) 
of LDC. (n = 35) 
District leaders are interested in spreading 0 (0.0%) 6 (17.1%) | 13 (37.1%) 2 (5.7%) 14 (40.0%) 
the use of LDC to additional schools. 
(i= 35) 
District professional development efforts 1 (2.9%) 10 (28.6%) | 17 (48.6%) 1 (2.9%) 6 (17.1%) 
were aligned with the LDC initiative. 
(n = 35) 
District leaders visited my school to discuss 3 (8.6%) 12 (34.3%) | 15 (42.9%) 1 (2.9%) 4 (11.4%) 
the implementation of LDC. (n = 35) 
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11 Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 


e CoreTools online platform 
e LDC online courses in the “Learn” section of CoreTools 
e Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e = /n-person coaching 


o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


A13. What supports did you find the most useful and why? 


A14. What supports were not helpful and why? 


A15. In what ways could LDC implementation be improved in your school in the future? 


177 


Appendix D: 
LDC Module Rating Dimensions 


Each module was rated on six dimensions. All of these were rated using a 5-point scale. With the first 
five, anchor points ranged from not present or realized to fully present or realized. In contrast, the 
overall dimension ranged from inadequate to advanced. 


Dimension 


1 Effective Writing Task 


Definition 

Degree to which teaching task makes effective use of the template task’s writing mode (i.e., argumentation or 
explanation); requires sustained writing and effective use of ideas and evidence to substantiate claims; and is 
feasible for most students to complete (i.e., appropriate for the grade-level and subject matter). 


Main Sources of Information 


Module Creator Handout (Task) 


e Read and evaluate the teaching task, student background/prior knowledge, and summary information. 
e Evaluate the difficulty or ease students may encounter trying to answer the question. 


e Compare module teaching task to teaching task template options. 


Anchor Points Description 


5 Fully Present or Realized The teaching task and performance expectations for the module are 
explicit and clear, require students to engage in higher-order thinking 
and writing, and are appropriate for the grade-level and subject matter. 


4  Sufficiently Present or Realized 


3 Moderately Present or Realized Clear module teaching task and performance expectations are 
available, but do not require students to engage in higher-order 
thinking and writing and/or are not appropriate for the grade-level and 
subject matter. 


2 ‘Barely Present or Realized 


1 Not Present or Realized There is minimal evidence of an effort to identify an explicit and clear 
teaching task and performance expectations that provide opportunities 
for students to think critically and are appropriate for the grade-level 
and subject matter. 
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Dimension 


Alignment to CCSS, Local, State Literacy and Content 
2 Standards 


Definition 
Extent to which module addresses content essential to the discipline, as well as reading comprehension and 
writing standards informed by local and state standards. 


Main Sources of Information 


Module Creator Handout (Task) 


e Read and evaluate the standards included in the module. 
e Module should include ELA as well as subject matter CCSS/state standards. 
e Compare and contrast the standards the module includes with those that could have been included. 


e Particular attention to content standards (CCSS History/Social Studies, Science, and Technical Subjects); 
State Standards; Specific Reading, Writing, Spoeaking/Listening, Language Skills 


Anchor Points Description 


5 Fully Present or Realized Module specifically addresses content essential to CCSS and local or 
state standards in science or social studies, as well as reading 
comprehension and writing. All standards are well aligned to the topic 
and teaching task. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized Module broadly addresses content essential to CCSS and local or state 
standards in science or social studies and reading comprehension and 
writing. Standards are sufficiently aligned to the topic and teaching 
task. 


2 ‘Barely Present or Realized 


1 Not Present or Realized Minimal evidence that module addresses content essential to the 
discipline and literacy standards. Standards are poorly aligned to the 
topic and teaching task. 
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Dimension 


e} Fidelity to LDC Module Instruction 


Definition 


Degree to which module instruction, activities, and teaching task address each of the four stages of instructional 
practice (preparation for the task, reading process, transition to writing, writing process). 


Main Sources of Information 


Module Creator Handout (Instruction) and Information Sheet 


e Evaluate for distribution of activities and time spent on each of the four stages of instructional practice. 


Anchor Points 


Description 


5 Fully Present or Realized 


The module instruction, activities, and teaching task reflect deliberate 
attention and fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect demonstrable effort to develop 
instructional scaffolding within and across each stage of instruction. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


The module instruction, activities, and teaching task reflect moderate 
attention and fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect sufficient effort to develop 
instructional scaffolding within and across each stage of instruction. 


2 ‘Barely Present or Realized 


1 Not Present or Realized 


The module instruction, activities, and teaching task reflect poor 
attention and lack of fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect inadequate effort to develop 
instructional scaffolding within and across each stage of instruction. 
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Dimension 


4 Quality Instructional Strategies 


Definition 

Degree to which the module provides clear instructional strategies aimed at helping students develop literacy skills 
and successfully complete the teaching task. In addition, the degree to which module instruction and activities 
scaffold critical thinking and performance in a way that is meaningful within the context of a given field or subject 
matter. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


e Evaluate extent to which the module activities scaffold critical thinking and student performance within 
the context of the subject matter at the core of the teaching task. 


e Evaluate extent to which instructional strategies guide student learning in literacy and ability to complete 
the teaching task. 


Anchor Points Description 


5 Fully Present or Realized Module provides clear and targeted instructional strategies and 
activities that scaffold student learning and promote critical thinking in 
social studies or science. There is explicit attention to helping students 
develop an accurate understanding of the topic and teaching task, and 
literacy skills necessary to complete the writing task successfully. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized Instructional strategies and activities are available to support adequate 
student learning and critical thinking in social studies or science. There 
is moderate attention to helping students develop an understanding of 
the topic and teaching task, and literacy skills necessary to complete 
the writing task. 


2 ‘Barely Present or Realized 


1 Not Present or Realized Limited instructional strategies and activities are available to support 
student learning and critical thinking in social studies or science. 
Insufficient attention to helping students develop an understanding of 
the topic and teaching task, or literacy skills necessary to complete the 
writing task. 
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Dimension 


5 Coherence and Clarity of Module 


Definition 
The degree of logical alignment found between the teaching task and the goals of the module with the readings, 
mini-tasks, and instructional strategies. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


Anchor Points Description 


5 Fully Present or Realized Strong alignment between the teaching task and goals of the module— 
including the CCSS and local and state literacy and content standards— 
with the readings, mini-tasks, student work, and instructional 
strategies. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized Moderate alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 


2 ‘Barely Present or Realized 


1 Not Present or Realized Poor alignment between the teaching task and goals of the module— 
including the CCSS and local and state literacy and content standards— 
with the readings, mini-tasks, student work, and instructional 
strategies. 
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Dimension 


3) Overall Impression 


Definition 
A holistic assessment of the LDC Module. 


Main Sources of Information 


Module Creator Handout, Classroom Handouts, and Student Work 


e To what extent does this module contribute to student college readiness and development of advanced 
literacy skills? 


Anchor Points 


Advanced LDC Module Implementation 


5 
4 Proficient LDC Module Implementation 


w 


Adequate LDC Module Implementation 


2 Marginal LDC Module Implementation 


1 Inadequate LDC Module Implementation 
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Appendix E: 
Research Procedure and Results for Principal Interviews 


To better understand teacher retention rates of LDC schools after the 2016-2017 school 
year, we conducted an interview study of school administrators. Twenty principals and assistant 
principals volunteered to be interviewed. This section presents the methods and results of the 
principal interviews. 


Method 
Instrument 


We developed a six-question interview protocol for principals and assistant principals. 
Some of the interview questions had subquestions. Five of the questions were asked of all 
principals, regardless of their schools’ retention typology, and one question (Question 5) 
differed slightly based on the school’s retention typology. See Figure E1 for the interview 
protocol. 
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Figure E1 


Principal and Assistant Principal Interview Protocol 


Question no. Question Given to 


il Why did you initially participate in LDC? All 
e What were your goals when you signed on and how well have those 
goals been met thus far? 
2 What was your role in LDC implementation? All 
e Who else at the school supported LDC? 
e How much did you see of PLC meetings? 
e Was part of your role to give feedback on LDC instruction? (If 


needed: Who gave feedback on LDC instruction?) 


3 Could you tell me about outside support — from both LDC and the All 
district — and what worked or didn’t work? 


4 What were the incentives for teachers’ participation in LDC? All 
e How would you describe teacher buy-in and commitment to LDC? 


e Did teachers receive any additional pay? If so, for what? 


5 What would you say are the main factors that led to not continuing with Drop out 
LDC? 


a) Looking across the district, we noticed that some schools chose to Zero 
continue with LDC for another year and others chose to stop. What retention 
would you say are the main factors that led to continuing with LDC? 


b) We noticed that none of the year-1 teachers participated in Year 2, 
so the year-2 PLC began with an entirely new set of teachers. Why 
do you think that was? 


c) Were there changes in implementation to keep this from 
happening again the following year? 


a) Looking across the district, we noticed that some schools chose to Very Low 
continue with LDC for another year and others chose to stop. What and Low 
would you say are the main factors that led to continuing with LDC? Retention 


b) We noticed that many teachers who participated in Year 1 did not 
participate in Year 2. Why do you think that was? 


a) Looking across the district, we noticed that some schools choseto Moderate- 
continue with LDC for another year and others chose to stop. What High 
would you say are the main factors that led to continuing with LDC? Retention 

b) We noticed that many teachers who participated in Year 1 
continued into Year 2. Why do you think that was? 


6 Do you have any advice for strengthening LDC or is there anything else _—Alll 
we should know about your experience with LDC? 
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Participants 


During the 2016-2017 school year, 48 total schools participated in the i3 LDC program 
including 28 in the New York City Department of Education (NYCDOE) and 20 in the West Coast 
school district which is the subject of this report. We categorized the schools into retention 
typologies based on a school’s total teacher participants in 2016-2017 and returning teachers 
in 2017-2018. Using the proportions of the same teacher participating for the 2 years, we 
created five school retention typologies: 


e dropout (the school has no teachers participating in LDC in 2017-2018); 


e zero retention (no 2016-2017 teachers returning to LDC, but new teachers in the 
school are participating in 2017-2018); 


e very low retention (less than 1/3 of teachers returning); 
e lowretention (between 1/3 and 2/3 returning teachers); and 
e moderate to high retention (over 2/3 teachers returning). 


Table E1 shows the number of schools per retention typology in NYCDOE and the West 
Coast district. 


Table E1 


Number of Schools and Proportion of Retention Typologies for NYCDOE and West Coast District in 
2016-2017 


# West Coast % West Coast 


# NYCDOE % NYCDOE district district 
Retention type schools schools schools schools 
Dropped out after 2016-2017 10 36% 6 30% 
Zero retention 1 4% 2 10% 
Very low teacher retention 7 25% 3 15% 
Low teacher retention 5 18% 2 10% 
Moderate to high teacher retention 5 18% 7 35% 
Total 28 20 


All LDC schools from the 2016-2017 cohort were contacted for a phone interview by 
email, and a total of 20 principals and assistant principals (10, or 36%, of NYCDOE schools and 
10, or 50%, of West Coast district schools) participated in the interview. Table E2 shows the 
school retention typologies that are represented in this sample. Of the principals and assistant 
principals who participated in an interview, most were from very low teacher retention schools 
(eight participants), followed by moderate to high teacher retention schools (six participants). 
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We also had three principals or assistant principals from dropout schools participate, two 
principals from low teacher retention schools, and one principal in a zero retention school. 


Table E2 
School Retention Typologies Represented in the Study 


# West Coast % West Coast 


# NYCDOE % NYCDOE district district 

Retention type schools schools schools schools 

Dropped out after 2016-2017 2 20% 1 17% 

Zero retention 0 0% 1 50% 

Very low teacher retention 5 71% 3 100% 

Low teacher retention 1 20% 1 50% 

Moderate to high teacher retention 2 40% 4 57% 
Total 10 10 


Procedures 


We originally contacted principals of all schools except for schools designated as low 
retention. This was done as sampling strategy for a case study methodology to understand 
reasons behind high and low retention schools. Initial emails were sent to 41 schools in both 
districts (23 in NYCDOE and 18 in the West Coast district) the week of June 16, 2018. A 
minimum of three follow-up emails for nonresponding administrators were sent between June 
25 and August 6, 2018. During this window, nine NYCDOE principals or assistant principals 
participated (39% of schools in the first round) and seven principals from the West Coast 
district participated (39% of schools in the first round). One West Coast district principal 
contacted us to say that they did not have enough knowledge about the LDC program in the 
school because they were the new principal. 


Because of the low response rate, we decided to contact the principals in the low teacher 
retention schools (five in NYCDOE and two in the West Coast district) to increase the numbers 
of participants. The initial emails for these NYCDOE principals were sent on September 20, 
2018, and for West Coast district principals, between October 1 and October 8, 2018. Follow-up 
emails were sent to NYCDOE and West Coast district principals between the weeks of October 1 
and October 19, 2018. We also reached out to the LDC grant directors in NYCDOE and the West 
Coast district for recruitment support. Between the new round of emails and support from the 
LDC grant directors, four additional school leaders participated in interviews, bringing our total 
participants to 20 principals and assistant principals (10, or 36%, of NYCDOE schools and 10, or 
50%, of West Coast district schools). 
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Interviews were scheduled at times most convenient to the principal or assistant 
principal. Interviews were conducted by two researchers following the protocol, and the 
interviews ranged from about eight minutes to 30 minutes in length. Participant consent was 
obtained prior to the interviews, and interviews were recorded. 


Analysis 


Recordings of interviews were transcribed. (For one interview, due to researcher and 
equipment error, the interview was not recorded. However, notes taken by the researcher 
during the interview were used for general counts in the analysis.) Transcriptions were read 
iteratively and coded for broad themes using grounded theory (Glaser & Strauss, 1967). 
Multiple close readings of each group transcript were conducted in order to summarize the 
data and identify themes related to reasons for a school’s retention (or lack thereof) of teachers 
from Year 1 to Year 2 of LDC implementation. A coding scheme was developed based on 
targeted areas of interest in relation to reasons for retention. 


Results 


Qualitative results from the data are presented in five main categories: reasons for 
teachers continuing in LDC, meeting goals, administrator’s role, teacher buy-in, and outside 
support. 


Reasons for Teachers Continuing or Leaving LDC in Year 2 


To understand why teachers remained or left LDC after Year 1, interview data were coded 
into three main categories: teacher or grade-level decision, administrative decision, and 
mandatory participation. In teacher or grade-level decisions, individual teachers, a grade level, 
or acontent-area team were free to decide to return the following year. Administrative 
decisions typically involved the principal deciding on changing the grade level or content-area 
teachers who would participate in the second year of implementation (e.g., moving from 
second-grade to third-grade teachers; dropping mathematics teachers but keeping science 
teachers in the program). A couple of principals also decided that LDC participation would be 
better suited for a particular level of teacher. For example, a principal decided to focus Year 2 
teacher participation on “teachers that were new to the profession and teachers that were 
struggling instructionally”; therefore, only a few teachers continued from Year 1 to Year 2 at 
that site. Mandatory participation, which is based on an administrator’s decision, reflected 
schools where teachers were told that participation in the second year of LDC was not 
voluntary. Table E3 shows the reasons for retention by school retention typology. 
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Table E3 
Teachers’ Participation in LDC in Year 2 by Retention Typology 


Teacher/grade Administrative Mandatory 

Retention typology level decision decision participation 
Dropped out after 2016-2017 1 2 0 
Zero retention 0 1 0 
Very low teacher retention 4 4 0 
Low teacher retention 1 0 1 
Moderate to high teacher retention 4 0 2 
Total 10 7 3 


For the three dropout schools in our sample, two of the schools had a change in 
leadership. One had a new principal who came in before the second year of LDC 
implementation and decided not to participate in LDC, and the other school had a new 
superintendent who did not continue partnering with LDC. For the last of the three dropout 
schools, the small group of teachers who were in the first year of implementation decided not 
to continue because of “teacher and administrator burn out.” This school had too many 
initiatives that were handled by a small number of administrative staff and the same teachers 
who were also part of the LDC implementation. 


For the one zero retention school in our sample, the principal decided to change the 
teachers for Year 2. The grade level that was selected to participate in LDC the first year found 
the program successful, and so for Year 2, “other teachers haven't been exposed, and we 
wanted more teachers to have access ... to spread the wealth within the school,” so another 
grade level was selected to participate. In one of the low retention schools, the principal said 
that participation in LDC was mandatory, but many of the teachers who were part of Year 1 
implementation left the school the following year. 


For the 10 schools that were coded as a teacher/grade-level decision, half of those 
decisions were based on the grade-level or content-area team (e.g., science teachers) deciding 
to leave LDC because they started a new program or initiative the second year and did not have 
the time to participate in LDC. For the moderate to high teacher retention schools who were 
coded as teacher/grade-level decision, the majority of teachers in those schools decided to stay 
in LDC the second year. 


Meeting Goals 


We asked participants their reasons for participating in LDC during the first year. The most 
popular reasons for participating in LDC were its alignment to standards and rigorous tasks. 
Principals and assistant principals also stated that they felt both teachers and students in their 
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schools needed assignments where literacy instruction, the writing tasks, and the writing 
process were a focus. Other responses included wanting better lesson planning, collaboration, 
and content-area instruction. Interestingly, three principals said that LDC appealed to them as 
they were new to the school or position. 


After stating their reasons for participation, we asked if these goals were met after the 
first year of implementation. Table E4 shows participant responses to whether goals were met 
by school typology. The majority of principals and assistant principals responded positively: 12 
(60%) said yes, five expressed mixed feelings about accomplishing their original goals for LDC 
(25%), and three said that their goals were not met (15%). From these responses, meeting goals 
for LDC did not appear to influence retention of teachers. For example, two of the three 
dropout schools reported meeting their LDC goals for participation, yet they didn’t remain in 
the program. Teacher retention seemed to be tied more to teacher and principal decisions (see 
section above). 


Table E4 
Principal Reponses for LDC Goals by School Typology 


LDC goals met 


School typology Yes No Mixed 
Dropped out after 2016-2017 2 uf 0 
Zero retention 1 0 0 
Very low teacher retention 4 1 3 
Low teacher retention 1 1 0 
Moderate to high teacher retention 4 0 2 
Total 12 3 5 


Administrator’s Role 


The majority of principals and assistant principals in our sample reported having an active 
role in supporting LDC implementation. Sixteen (80%) attended LDC PLCs, and those who could 
not make the PLCs were able to get a summary of the meetings from their assistant principals 
or teacher leaders. For those who were able to attend some PLCs, the amount of time they 
were able to sit in on the PLCs varied. Some principals were only able to drop in for a portion of 
the PLCs. For example, one principal (moderate to high retention school) said, “I would 
probably just go in the last 15 minutes. | don't think | was in a full session all the time. But | 
would touch bases and then each teacher would give me a wrap up summary of what they had 
done, what they were working on." Others were able to go to almost all PLCs. As the zero 
retention school principal said, “| was a participant, also, in the learning.” Sixteen participants 
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(80%) also reported that they gave feedback on LDC instruction. However, the degree to which 
principals or assistant principals gave feedback varied. Table ES shows the numbers of 
participants who attended PLCs and provided feedback on LDC instruction. 


Table E5 
Participation of Principals and Assistant Principals in LDC Implementation by Retention 
Type 


Retention typology Attended PLCs Provided feedback 
Dropped out after 2016-2017 3 3 
Zero retention 1 1 
Very low teacher retention 6 6 
Low teacher retention 1 2 
Moderate to high teacher retention 5 4 
Total 16 16 


Teacher Buy-in 


Most of the principals and assistant principals mentioned increasing skills and knowledge, 
collaboration, and student achievement and accessing LDC materials as incentives for teachers. 
Besides these incentives that were actually part of the LDC implementation, a few schools were 
able to provide additional school-based supports. Eight schools (40%) in our sample reported 
providing teachers with more tangible incentives for their participation in LDC, such as 
additional pay and time, and four of the eight gave multiple incentives. Table E6 shows the 
number of these types of teacher incentives by school typology. No pattern emerges that 
connects teacher incentives to retention typology. However, it is notable that the majority of 
schools in the sample did not provide tangible incentives (i.e., pay, extra time, teacher 
evaluation) to teachers for their participation for LDC. 
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Table E6 
Teacher Incentives by School Typology 


Retention typology Additional pay —_ Extra time Teacher evaluation 
Dropped out after 2016-2017 0 0 0 
Zero retention 1 1 0 
Very low teacher retention 2 0 1 
Low teacher retention 1 2 0 
Moderate to high teacher retention 1 1 1 
Total 5 4 2 


Five schools (25%) were able to pay teachers for their participation in LDC. Aside from 
pay, some principals were able to support participating teachers with extra time, such as 
adjusting the schedule to give teachers more time to meet during the week or giving teachers a 
release day. For example, the principal from the zero retention school said, “Initially, it was kind 
of an imposition of time, but when we created extra time and space for the teachers to meet, 
they were appreciative. So they had more buy-in.” Another principal (low retention school) 
said, 

| make it very easy for teachers, because I've built in time in their 
schedule for them to meet for LDC. There are no discrepancies in terms 
of when we should meet or how can we meet, or what needs to be done, 
which is usually the issue sometimes when it comes to work with outside 
consultants or providers, right? 


Two schools incentivized participation through teacher evaluations. One principal (very 
low retention school) postponed teacher evaluations for the year. The other principal 
(moderate to high retention school) said, “I very much tied it to teacher performance and my 
expectation of what high quality planning looks like.” For this principal, they knew LDC would 
be 


transformative. | knew that if they got through it and actually did it, the 
carrot would reveal itself in the students’ performance. And | knew my 
teachers were committed; | knew they wanted our kids to do well. They 
just needed to be pushed through the process to see how to get the kids 
to do well. 


Outside Support 


Overwhelmingly positive responses were reported from our sample regarding LDC 
support the schools received. All principals and assistant principals mentioned either training or 
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coaching that LDC provided. Two principals, both from very low retention schools, also 
mentioned that the administrator meetings were helpful. As one principal described, “It gave 
an opportunity to speak to other people who were in the project, and it also gave an 
opportunity to actually go through all of the CoreTools that were there to support the 
teachers.” Two principals (both from New York) also found site visits to another school helpful: 
“There are sites where the work is going on really nicely, and we were able to go over to those 
sites and see presentations from the principal and her teachers on how they have embedded 
this entire system into their daily program.” Five participants mentioned district support being 
available. 


Discussion 


Participants in this study all reported playing an active role in LDC implementation at their 
school site, and most said that their goals for participation were met. They also overwhelmingly 
responded favorably to LDC support, and district support, if provided or perceived, was also 
helpful. The principal interviews showed the diversity of reasons for why schools were able to 
retain teachers, and they also show why teachers left. The main reason for teacher retention 
was teacher and grade-level team decisions to leave or stay, followed by the principals’ decision 
to switch participation of teachers from Year 1 to Year 2. Teacher buy-in also seemed to affect 
teachers’ decisions for remaining or continuing. We did not find any consistent trends for 
retention typology; that is, we cannot characterize reasons for each retention typology and say, 
for example, all dropout schools had these characteristics or most moderate to high retention 
schools had other characteristics. Teacher retention in LDC from Year 1 to Year 2 was due toa 
myriad of factors that worked in combination at each school. 
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Appendix F: 
Additional Results on Module Artifact Ratings 


The following presents further details of the module analyses. First we present the 
methodology and results of the generalizability theory studies. This is followed by the 
additional tables for the descriptive analyses. 


Generalizability Study 


Generalizability theory is a statistical framework for examining multiple sources of 
potential error during the scoring process. For each grade band, we first modeled score 
variability across all six dimensions using a two-faceted design, whereby we estimated variance 
components for module by rater by dimension (t*r*d). The goal here was to separate true 
variation in the modules from other potential sources of measurement error. The main effects 
reflect true variation across modules (o2t) and error variance across raters (o2r) and 
dimensions (o2d), while the error term (o2trd,e) reflects unexplained residual error in the 
model. To disentangle the sources of potential error further, we also used a single-faceted 
design to examine potential error within the scoring of each dimension. As with the first set of 
models, the main effect reflects true variation across teachers (o2t) and error variance across 
raters (o2r). 


Elementary Module Results 


Generalizability theory models were conducted to examine potential error in the scoring 
process for the elementary modules. Results from the two-faceted and one-faceted models 
that examine error across and within dimensions are presented in Tables F1 and F2. As would 
be the goal of any rating session, most of the variation found for the elementary modules was 
due directly to differences in the modules (57%) or to differences in the modules by dimension 
(35%). Despite this, 7% of the variation was due either directly or through interaction with the 
raters. Finally, only 0.09% of the variation was unexplained by the two-faceted model used. 


As previously mentioned, we also used a one-faceted design to disentangle variation in 
the ratings that was due either directly or through interaction with the dimensions (see 
Table F2). As would be desired, the greatest source of variation for each dimension was due 
directly to differences in the modules. Despite this, moderate amounts of variation could be 
attributed either directly or through interaction with the raters for two of the dimensions. More 
specifically, with Dimension 4, which measures the quality of the instructional strategies used, 
9% of the variation was due to the raters and 24% was due to an interaction between raters 
and modules. In addition, with Dimension 6, which measures the raters’ overall impression of 
the module, 19% of the variation was due either directly or through interaction with the raters. 
Finally, there was no unexplained error variance detected by the models. 


194 


Table F1 


Generalizability Study of the Elementary Module Ratings Across Dimensions (n = 139) 


Source Var % 
Module (o2t) 0.73 57.17 
Rater (o2r) 0.05 4.07 
Dimension (02d) 0.01 0.39 
Module x Dimension (o2td) 0.45 35.15 
Rater x Dimension (o2rd) 0.02 1.22 
Module x Rater (o2tr) 0.02 1.90 
Error (o2trd,e) 0.00 0.09 


Table F2 


Generalizability Study of the Elementary Module Ratings for Each Dimension (n = 139) 


Module (o2t) 


Rater (o2r) 


Module x 
Rater (o2tr) 


Error (o2trd,e) 


Dimension Var. % Var. % Var. % Var. % 
1. Effective writing task 0.99 90.94 0.05 4.83 0.00 0.00 0.00 0.00 
2. Standards alignment 1.39 94.32 0.05 3.48 0.03 2.20 0.00 0.00 
3. Fidelity to LDC instruction 1.36 94.12 0.05 3.67 0.00 0.00 0.00 0.00 
4. Quality instructional strategies 1.12 66.82 0.16 9.26 0.40 23.92 0.00 0.00 
5. Coherence/clarity of module 1.15 87.82 0.07 5.49 0.00 0.00 0.00 0.00 
6. Overall impression 0.92 80.87 0.15 5.83 0.15 13.30 0.00 0.00 


Note. Negative estimates of variance were changed to zero in order to calculate percentages (see 


Shavelson & Webb, 1991). 


Secondary Module Results 


Generalizability theory models were conducted to examine potential error in the scoring 
process for the secondary modules. Tables F3 and F4 present results from the two-faceted and 
one-faceted models that examine error across and within dimensions. As would be the goal of 


any rating session, most of the variation found in the ratings was due directly to differences in 
the modules (38%) or to differences in the modules by dimension (48%). Furthermore, only 7% 
of the variation for the secondary modules was due either directly or indirectly to the raters. 
What is of concern, though, is that 4% of the variation in ratings for the two-faceted model was 


unexplained. 


195 


Table F3 
Generalizability Study of the Secondary Module Ratings Across Dimensions (n = 66) 


Source Var. % 
Module (o2t) 0.37 38.44 
Rater (o2r) 0.00 0.00 
Dimension (02d) 0.02 1.89 
Module x Dimension (o2td) 0.47 48.27 
Rater x Dimension (o2rd) 0.05 5.21 
Module x Rater (o2tr) 0.02 2.16 
Error (o2trd,e) 0.04 4.03 


We also used a one-faceted design to disentangle the variance that was due either 
directly or through interaction with the dimensions (see Table F4). As would be desired, 77% to 
90% of the variance found for each dimension was attributable to differences in the modules. 
Furthermore, only two dimensions had moderate amounts of variance due directly to the raters 
(Dimension 2 = 12%, Dimension 5 = 9%). The only concerns involved the moderate amounts of 
variance due to an interaction between modules and raters for the dimensions (8% to 20%), 
with the greatest amount being found for Dimension 6, which measures the raters’ overall 
impression of module quality. Finally, there was no unexplained error variance found for any of 
the one-faceted models. 


Table F4 
Generalizability Study of the Secondary Module Ratings for Each Dimension (n = 66) 


Module x 
Module (o2t) Rater (o2r) Rater (o2tr) Error (o2trd,e) 
Dimension Var. % Var. % Var. % Var. % 
1. Effective writing task 0.73 89.96 0.00 0.00 0.08 10.04 0.00 0.00 
2. Standards alignment 0.60 77.44 0.09 11.78 0.08 10.79 0.00 0.00 
3. Fidelity to LDC instruction 1.00 87.71 0.00 0.00 0.14 12.29 0.00 0.00 


4. Quality instructional strategies 0.92 84.53 0.01 0.99 0.16 14.49 0.00 0.00 
5. Coherence/clarity of module 0.85 83.61 0.09 8.62 0.08 7.77 0.00 0.00 
6. Overall impression 0.56 79.84 0.00 0.00 0.14 20.16 0.00 0.00 


Note. Negative estimates of variance were changed to zero in order to calculate percentages (see 
Shavelson & Webb, 1991). 
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Summary 


Generalizability models were fit for the overall samples for the two grade bands. As would 
be hoped for, when examining the one-faceted models, the majority of variation for the 
elementary and secondary ratings were due directly to differences in the modules (57%, 38%) 
or to differing quality in the modules across dimensions (35%, 48%). Likewise, results from the 
two-faceted models showed that the majority of variation within dimension was due to 
differences in both the elementary modules (67% to 94%) and secondary modules (77% to 
90%). Despite this, for the elementary modules 24% of the variation for Dimension 4, which 
measures quality instructional strategies, and 13% of the variation for Dimension 6, which 
measures overall quality, were due to an interaction between the raters and modules. In 
contrast, variation due to an interaction between raters and the secondary modules ranged 
from 8% to 20% depending upon the dimension, with the greatest amount found for Dimension 
6. 


Descriptive Results 


The following section presents percentages for the elementary and secondary modules. 
This includes the overall results, and then results broken down by content area and then cohort 


group. 
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Table F5 


Distribution (Percentage) of Ratings for the Modules 


Dimension 1 2 3 4 5 
Elementary (n = 104) 
1. Effective writing task 2.9 13.5 35.6 32.7 15.4 
2. Standards alignment 8.7 15.4 31.7 28.8 15.4 
3. Fidelity to LDC instruction 11.5 24.0 27.9 26.0 10.6 
4. Quality instructional strategies 9.6 17.3 32.7 26.0 14.4 
5. Coherence/clarity of module 7.7 18.3 33.7 26.0 14.4 
6. Overall impression 2.9 19.2 40.4 25.0 10.6 
Secondary (n = 51) 
1. Effective writing task 0.0 7.8 29.4 35.3 27.5 
2. Standards alignment 0.0 3.9 37.3 33.3 25.5 
3. Fidelity to LDC instruction 5.9 11.8 29.4 37.3 15.7 
4. Quality instructional strategies 5.9 5.9 23.5 43.1 21.6 
5. Coherence/clarity of module 3.9 7.8 29.4 41.2 17.6 
6. Overall impression 2.0 5.9 37.3 41.2 13.7 
Overall (n = 155) 
1. Effective writing task 1.9 11.6 33.5 33.5 19.4 
2. Standards alignment 5.8 11.6 33.5 30.3 18.7 
3. Fidelity to LDC instruction 9.7 20.0 28.4 29.7 12.3 
4. Quality instructional strategies 8.4 13.5 29.7 31.6 16.8 
5. Coherence/clarity of module 6.5 14.8 32.3 31.0 15.5 
6. Overall impression 2.6 14.8 39.4 30.3 11.6 
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Table F6 


Distribution (Percentage) of Ratings for the Elementary Modules by Content Area 


Dimension 1 2 3 4 5 
ELA (n = 38) 
1. Effective writing task 0.0 10.5 44.7 36.8 7.9 
2. Standards alignment 0.0 13.2 55.3 23.7 7.9 
3. Fidelity to LDC instruction 0.0 21.1 39.5 23.7 15.8 
4. Quality instructional strategies 0.0 15.8 34.2 34.2 15.8 
5. Coherence/clarity of module 0.0 15.8 28.9 36.8 18.4 
6. Overall impression 0.0 15.8 44.7 26.3 13.2 
Science (n = 43) 
1. Effective writing task 0.0 11.6 27.9 39.5 20.9 
2. Standards alignment 11.6 9.3 11.6 46.5 20.9 
3. Fidelity to LDC instruction 16.3 18.6 23.3 32.6 9.3 
4. Quality instructional strategies 11.6 9.3 39.5 23.3 16.3 
5. Coherence/clarity of module 9.3 14.0 39.5 20.9 16.3 
6. Overall impression 0.0 18.6 41.9 27.9 11.6 
Social studies (n = 23) 
1. Effective writing task 13.0 21.7 34.8 13.0 17.4 
2. Standards alignment 17.4 30.4 30.4 4.3 17.4 
3. Fidelity to LDC instruction 21.7 39.1 17.4 17.4 4.3 
4. Quality instructional strategies 21.7 34.8 17.4 17.4 8.7 
5. Coherence/clarity of module 17.4 30.4 30.4 17.4 4.3 
6. Overall impression 13.0 26.1 30.4 17.4 4.3 
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Table F7 


Distribution (Percentage) of Ratings for the Elementary Modules by Cohort 


Dimension 1 2 3 4 5 

Cohort1, 2016-2017 (n = 16) 

1. Effective writing task 6.3 18.8 6.3 56.3 12.5 
2. Standards alignment 12.5 0.0 18.8 50.0 18.8 
3. Fidelity to LDC instruction 25.0 6.3 31.3 31.3 6.3 
4. Quality instructional strategies 18.8 6.3 37.5 25.0 12.5 
5. Coherence/clarity of module 18.8 12.6 25.0 31.3 12.5 
6. Overall impression 0.0 6.3 12.5 37.5 31.3 
Cohort 2, 2017-2018 (n = 51) 

1. Effective writing task 2.0 9.8 45.1 27.5 15.7 
2. Standards alignment 9.8 15.7 41.2 19.6 13.7 
3. Fidelity to LDC instruction 9.8 17.6 37.3 23.5 11.8 
4. Quality instructional strategies 7.8 13.7 39.2 23.5 15.7 
5. Coherence/clarity of module 5.9 17.6 37.3 27.5 11.8 
6. Overall impression 0.0 3.9 17.6 47.1 23.5 
Cohort 2, 2018-2019 (n = 30) 

1. Effective writing task 3.3 13.3 33.3 30.0 20.0 
2. Standards alignment 6.7 13.3 23.3 36.7 20.0 
3. Fidelity to LDC instruction 10.0 33.3 13.3 30.0 13.3 
4. Quality instructional strategies 10.0 16.7 23.3 33.3 16.7 
5. Coherence/clarity of module 6.7 13.3 33.3 23.3 23.3 
6. Overall impression 3.3 0.0 20.0 33.3 26.7 


Note. Cohort groups with less than five are not presented. 
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Table F8 


Distribution (Percentage) of Ratings for the Secondary Modules by Content Area 


Dimension 1 2 3 4 5 
ELA (n = 32) 
1. Effective writing task 0.0 9.4 18.8 37.5 34.4 
2. Standards alignment 0.0 6.3 34.4 28.1 31.3 
3. Fidelity to LDC instruction 6.3 15.6 28.1 28.1 21.9 
4. Quality instructional strategies 6.3 6.3 28.1 28.1 31.3 
5. Coherence/clarity of module 6.3 9.4 31.3 28.1 25.0 
6. Overall impression 3.1 9.4 37.5 31.3 18.8 
Science (n = 3) 
1. Effective writing task _ — _ _ _ 
2. Standards alignment _ _ _ _ — 
3. Fidelity to LDC instruction — — — _ — 
4. Quality instructional strategies _ _ _ _ — 
5. Coherence/clarity of module — _ _ _ _ 
6. Overall impression — _ _ —_ _ 
Social studies (n = 16) 
1. Effective writing task 0.0 6.3 50.0 31.3 12.5 
2. Standards alignment 0.0 0.0 50.0 43.8 6.3 
3. Fidelity to LDC instruction 6.3 6.3 25.0 56.3 6.3 
4. Quality instructional strategies 6.3 0.0 12.5 75.0 6.3 
5. Coherence/clarity of module 0.0 0.0 18.8 75.0 6.3 
6. Overall impression 0.0 0.0 31.3 62.5 6.3 


Note. Content areas with less than five are not presented. 
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Table F9 
Distribution (Percentage) of Ratings for the Secondary Modules by Cohort 


Dimension 1 2 3 4 5 


Cohort 1, 2017-2018 (n = 6) 


1. Effective writing task 0.0 33.3 0.0 66.7 0.0 
2. Standards alignment 0.0 0.0 16.7 50.0 33.3 
3. Fidelity to LDC instruction 0.0 33.3 50.0 16.7 0.0 
4. Quality instructional strategies 0.0 0.0 16.7 66.7 16.7 
5. Coherence/clarity of module 0.0 0.0 16.7 66.7 16.7 
6. Overall impression 0.0 0.0 50.0 50.0 0.0 


Cohort 2, 2017-2018 (n = 26) 


1. Effective writing task 0.0 3.8 30.8 30.8 34.6 
2. Standards alignment 0.0 7.7 38.5 30.8 23.1 
3. Fidelity to LDC instruction 3.8 11.5 38.5 30.8 15.4 
4. Quality instructional strategies 3.8 7.7 30.8 34.6 23.1 
5. Coherence/clarity of module 3.8 7.7 34.6 34.6 19.2 
6. Overall impression 0.0 3.8 50.0 30.8 15.4 


Cohort 2, 2018-2019 (n = 13) 


1. Effective writing task 0.0 7.7 53.8 23.1 15.4 
2. Standards alignment 0.0 0.0 38.5 46.2 15.4 
3. Fidelity to LDC instruction 7.7 7.7 15.4 46.2 23.1 
4. Quality instructional strategies 7.7 7.7 7.7 61.5 15.4 
5. Coherence/clarity of module 0.0 15.4 15.4 61.5 7.7 
6. Overall impression 0.0 15.4 15.4 61.5 7.7 


Note. Cohort groups with less than five are not presented. 
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Table F10 


Means and Standard Deviations for the Exploratory Analysis of Elementary Modules by Cohort 


Dimension 2017-2018 2018-2019 Change 

Cohort 1 (n = 9) 

1. Effective writing task 3.78 (0.83) 2.89 (0.93) -0.89 (0.93) 
2. Standards alignment 3.89 (0.78) 3.00 (1.22) -0.89 (1.27) 
3. Fidelity to LDC instruction 3.44 (1.33) 2.89 (1.27) -0.56 (1.33) 
4. Quality instructional strategies 3.00 (1.66) 3.00 (1.22) 0.00 (1.94) 
5. Coherence/clarity of module 3.22 (1.48) 3.11 (1.27) -0.11 (1.05) 
6. Overall impression 3.22 (1.64) 3.22 (0.97) 0.00 (1.41) 
Average (Dimensions 1 to 5) 3.47 (1.08) 2.98 (0.94) -0.49 (0.81) 
Cohort 2 (n = 31) 

1. Effective writing task 4.39 (0.72) 3.52 (0.85) -0.87 (1.12) 
2. Standards alignment 3.97 (0.87) 3.23 (1.26) -0.74 (1.59) 
3. Fidelity to LDC instruction 4.10 (1.25) 3.23 (1.23) -0.87 (1.06) 
4. Quality instructional strategies 4.10 (1.11) 3.52 (1.06) -0.58 (1.34) 
5. Coherence/clarity of module 4.03 (0.98) 3.55 (0.96) -0.48 (1.03) 
6. Overall impression 4.00 (1.00) 3.42 (0.89) -0.58 (1.12) 
Average (Dimensions 1 to 5) 4.12 (0.85) 3.41 (0.85) -0.71 (0.98) 
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Table F11 


Means and Standard Deviations for the Exploratory Analysis of Secondary Modules by Cohort 


Dimension 2017-2018 2018-2019 Change 

Cohort 1 (n = 7) 

1. Effective writing task 4.14 (0.90) 4.00 (1.00) -0.14 (1.46) 
2. Standards alignment 3.71 (0.95) 4.43 (0.79) 0.71 (1.38) 
3. Fidelity to LDC instruction 4.00 (0.82) 3.29 (0.76) -0.71 (1.11) 
4. Quality instructional strategies 4.00 (1.00) 4.00 (0.82) 0.00 (0.82) 
5. Coherence/clarity of module 3.57 (1.90) 4.00 (0.82) 0.43 (1.40) 
6. Overall impression 4.00 (0.58) 3.86 (0.69) -0.14 (0.90) 
Average (Dimensions 1 to 5) 3.89 (0.65) 3.94 (0.59) 0.06 (0.81) 
Cohort 2 (n = 20) 

1. Effective writing task 3.40 (1.27) 4.05 (0.94) 0.65 (1.42) 
2. Standards alignment 3.40 (0.88) 3.70 (0.92) 0.30 (1.38) 
3. Fidelity to LDC instruction 3.60 (1.14) 3.30 (1.03) -0.30 (1.45) 
4. Quality instructional strategies 3.80 (1.11) 3.70 (1.17) -0.10 (1.37) 
5. Coherence/clarity of module 3.90 (0.79) 3.70 (0.92) -0.20 (1.01) 
6. Overall impression 3.75 (0.85) 3.65 (0.81) -0.10 (1.17) 
Average (Dimensions 1 to 5) 3.62 (0.76) 3.69 (0.69) 0.07 (0.93) 
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Table F12 
Distribution (Percentage) of Ratings for the Exploratory Analysis of Elementary Modules by Cohort 


Negative No change Positive 
Dimension # % # % # % 

Cohort 1 (n = 9) 

1. Effective writing task 7 77.8 1 11.1 1 11.1 
2. Standards alignment 5 55.6 3 33.3 1 11.1 
3. Fidelity to LDC instruction 5 55.6 1 11.1 3 33.3 
4. Quality instructional strategies 3 33.3 1 11.1 5 55.6 
5. Coherence/clarity of module 3 33.3 3 33.3 3 33.3 
6. Overall impression 3 33.3 4 44.4 2 22.2 
Average (Dimensions 1 to 5) 5 55.6 2 22.2 2 22.2 
Cohort 2 (n = 31) 

1. Effective writing task 23 74.2 4 12.9 4 12.9 
2. Standards alignment 15 48.4 10 32.3 6 19.4 
3. Fidelity to LDC instruction 17 54.8 12 38.7 2 6.5 
4. Quality instructional strategies 19 61.3 5 16.1 7 22.6 
5. Coherence/clarity of module 17 54.8 7 22.6 7 22.6 
6. Overall impression 18 58.1 7 22.6 6 19.4 
Average (Dimensions 1 to 5) 22 71.0 3 So 6 19.4 
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Table F13 
Distribution (Percentage) of Ratings for the Exploratory Analysis of Secondary Modules by Cohort 


Negative No change Positive 
Dimension # % # % # % 

Cohort 1 (n = 7) 

1. Effective writing task 4 57.1 0 0.0 3 42.9 
2. Standards alignment 2 28.6 1 14.3 4 57.1 
3. Fidelity to LDC instruction 3 42.9 4 57.1 0 0.0 
4. Quality instructional strategies 2 28.6 3 42.9 2 28.6 
5. Coherence/clarity of module 2 28.6 2 28.6 3 42.9 
6. Overall impression 3 42.9 2 28.6 2 28.6 
Average (Dimensions 1 to 5) 4 57.1 0 0.0 3 42.9 
Cohort 2 (n = 20) 

1. Effective writing task 5 25.0 4 20.0 11 55.0 
2. Standards alignment 6 30.0 5 25.0 9 45.0 
3. Fidelity to LDC instruction 7 35.0 9 45.0 4 20.0 
4. Quality instructional strategies 7 35.0 7 35.0 6 30.0 
5. Coherence/clarity of module 8 40.0 6 30.0 6 30.0 
6. Overall impression 8 40.0 6 30.0 6 30.0 
Average (Dimensions 1 to 5) 8 40.0 2 10.0 10 50.0 
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Appendix G: 


Fidelity Matrix 


1. Key Component = Common Planning time for LDC Professional Learning Community with Synchronous Coach Support 


Teacher Expectation that 
Attendance at |PLC teachers will 
weekly PLC regularly attend 
meetings PLC meetings. 


Teacher 


PLC 
Reflection 
form will 
include 
attendance 
record for 
both coach- 
facilitated 
and teacher 
leader- 
facilitated 
PLC 
meetings 


Reflection form 
will be filled out 
by Teacher 
Leader for 
each weekly 
PLC session. 
LDC will 
deliver 
attendance 
data to 
CRESST twice 
per year. 


0 (very low) = less 
than 70% 
attendance at PLC 
sessions 

1 (low) = 70-79% 
attendance at PLC 
sessions 

2 (moderate) = 80- 
89% attendance at 
PLC sessions 

3 (ideal) = at least 
90% attendance at 
PLC sessions 


Adequate 
implementation at 
teacher level is score 
of 2 


School level: 

0 (very low) = less 
than 65% of PLC 
teachers in school 
with score of 2 

1 (low) = 65% to 
<75% of PLC 
teachers in school 
with score of 2 

2 (moderate) = 75- 
85% of PLC teachers 
in school with score 
of 2 

3 (high) = over 85% 
of PLC teachers in 
school with score of 2 


Adequate 
implementation at 
school level is score 
of 2. 


Sample-level 

0= < 25% schools with 
score = 2 

1 = 26-50% schools with 
score 22 

2 = 51-75% schools with 
score = 2 

3 = > 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-2018 to 
2018-2019 
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Amount of time 
spent on LDC 
during common 
planning time 


PLCs expected to 
spend at least 60 
minutes per PLC 
meeting where the 
coach joins. 


School 


PLC 
reflection 
form 


Teacher 
Leader will 
note time spent 
on LDC in the 
same reflection 
form that 
captures 
attendance. 
Data delivered 
to CRESST 
twice per year. 


0 (low) = modal* 
response of under 
45 minutes per 
push-in session 

1 (moderate) = 
modal response of 
45-59 minutes per 
push-in session 

2 (high) = modal 
response of 60-74 
minutes per push-in 
session 

3 (ideal) = modal 
response of 75-90 
minutes per push-in 
session 


*if there is more 
than one modal 
response, the 
highest modal 
response will be 
used. 


Adequate 
implementation at 
school level is score 
of 2 


Sample level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 = > 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-2018 to 
2018-2019 
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Exposure to 
LDC LEARN 
content during 
first 
instructional 
cycle 


Coaches are 
expected to use 
Instructional Cycles 
in their biweekly 
digital push-in 
sessions with the 
PLCs. 


Teacher 


CoreTools 
Data 


Users view 
Instructional 
Cycle sessions 
during coach 
push-in 
sessions. Data 
delivered to 
CRESST twice 
per year. 


0 (very low) = 
teacher views less 
than 50% of 
sessions in the 
Instructional Cycle 
1 (low) = teacher 
views between 
50% and less than 
60% of sessions in 
the Instructional 
Cycle 

2 (moderate) 
teacher views 
between 60% and 
less than 70% of 
sessions in the 
Instructional Cycle 
3 (high) teacher 
views 70% or more 
of sessions in the 
Instructional Cycle 


Adequate 
implementation at 
school level is score 
of 2. 


School level: 0 (very 
low) = less than 65% 
of PLC teachers in 
school with score of 2 
1 (low) = 65% to 
<75% of PLC 
teachers in school 
with score of 2 2 
(moderate) = 75-85% 
of PLC teachers in 
school with score of 2 
3 (high) = over 85% 
of PLC teachers in 
school with score of 2 
Adequate 
implementation at 
school level is score 
of 2. 


Sample level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score 2 2 

3 = > 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-2018 to 
2018-2019 
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Exposure to 
LDC LEARN 
content during 
second 
instructional 
cycle 


Coaches are 
expected to use 
Instructional Cycles 
in their biweekly 
digital push-in 
sessions with the 
PLCs. 


Teacher 


CoreTools 
Data 


Users view 
Instructional 
Cycle sessions 
during coach 
push-in 
sessions. Data 
delivered to 
CRESST twice 
per year. 


0 (very low) = 
teacher views less 
than 50% of 
sessions in the 
Instructional Cycle 
1 (low) = teacher 
views between 
50% and less than 
60% of sessions in 
the Instructional 
Cycle 

2 (moderate) 
teacher views 
between 60% and 
less than 70% of 
sessions in the 
Instructional Cycle 
3 (high) teacher 
views 70% or more 
of sessions in the 
Instructional Cycle 


Adequate 
implementation at 
school level is score 
of 2. 


School level: 0 (very 
low) = less than 65% 
of PLC teachers in 
school with score of 2 
1 (low) = 65% to 
<75% of PLC 
teachers in school 
with score of 2 2 
(moderate) = 75-85% 
of PLC teachers in 
school with score of 2 
3 (high) = over 85% 
of PLC teachers in 
school with score of 2 
Adequate 
implementation at 
school level is score 
of 2. 


Sample level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score 2 2 

3 = > 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-2018 to 
2018-2019 
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1 = adequate 
implementation for each 
indicator 

0 = inadequate 
implementation for one or 
more indicator 


Adequate implementation 
at sample level for key 
component is score of 1 


Perceived Common planning |Teacher Teacher Teacher 0 (very low) = Adequate School level: Sample level All participating | Once per year from 
effectiveness of | time is expected to survey Survey asks _| teacher reports that | implementation is 0 (verylow)=less |0=<25% schools with | Schools 2016-2017 to 
engagementin |lead to teacher respondents to |on average her score of 2 at teacher |than 65% of PLC score > 2 2018-2019 
PLC on teacher | proficiency in 4 key report abilities did not level teachers in school 4 = 26-50% schools with 
competencies competencies: improvement | improve at all with score of 2 seared 
identifying from the between the 4 (low) = 65% t 2 ; 
standards-aligned beginning to —_| beginning and end Hide Le : ae ey schools with 
assignments, the end of the of the year’s work teachers in school eduaien : ; 
construction of year. There are | with LDC vin cenheor? 3 = > 75% schools with 
quality assignment 7 items 1 (low) = teacher 9 (moderate) = 75 score 2 2 
prompts, covering the 4 | reports that on her see enna 
developing competencies. abilities improved a ih pre serwileeers Adequate implementation 
ba aes al me ial he of 2 | ome level is a score 
moderate) = igh) = 
assessment. measure are lie fs ee that 3 (high) = over 85% 
based on an sda of PLC teachers in 
eens her abilities ; 
index improved school with score of 2 
averaging the moderately 
seven items. | (high) = teacher st ae 
reports her abilities Ce at 4 : poll 
improved a great oo evel is score 
deal ae: 
Allindicators |N/A N/A N/A N/A N/A N/A N/A Sample-level for key All participating | Once per year from 
component as awhole _| schools 2017-2018 to 
2018-2019 
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sequence: teaching 
task, complete 
module, and 
revised module. 


with 
information on 
teachers’ 
interaction with 
the online 
courses and 
CoreTools 
features 


1 (moderate) = 
coach feedback 
provided on a 
linked course once 
via peer review 
rubric per year per 
teacher 


teachers from school 
PLC with score of 1 
2 (moderate) = 75- 
85% of teachers from 
school PLC with 
score of 1 

3 (high) = over 85% 
of teachers from 
school PLC with 
score of 1 Adequate 
implementation at 


score 2 2 


3 = > 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 2 


Coach Coaches required | Module CoreTools |LDC.org will |0(verylow)=no |Adequate School level: 0 (very |Sample-level All participating | Once per year from 
Comments on _|to provide feedback | (universe is _|Analytic provide at least |coach comments __|implementation at low) = less than 65% | = < 25% schools with | Schools 2017-2018 to 
Modules on modules at set |all modules |data 2 data pulls per) 4 (low) = coach module level is score |of modules from score 22 2018-2019 
points in the linked to year of comment provided |of 2 school PLC with 4 = 26-50% schools with 
pablo r LEARN - ee | at one key point in score of 2 scored 
sequence: teaching |courses eacher level | design 1 (low) = 65-74% of ||» = 51.75% schools with 
task, complete CoreTools data] > (moderate) = modules from school |oonres 5. ” 
eae Wtormation on (coach comment PEC wih score of 2 | = > 75% schools with 
revised module. rie °N | provided at two key 2 (moderate) =75- | ooore > . 
teachers —__|points in design 85% of modules from 7 
interaction with 3 (high) = coach school PLC with 
the online , a provided score of 2 Adequate implementation 
pean at three or more 3 (high) = over 85% : al Ivenle aiseote 
feobinas key points in design of modules from : 
school PLC with 
score of 2 Adequate 
implementation at 
school level is score 
of 2. 
Coach Coaches required |Teacher CoreTools |LDC.org will 0 (low) = coach Adequate School level: Sample-level All participating | Once per year from 
Formative Peer |to provide feedback |(universe is | Analytic provide at least | feedback not implementation at O(verylow)=less |0=<25% schools with |Schools 2017-2018 to 
Review on on modules via all modules | data 2 data pulls per | provided on a teacher level is score |than 65% of teachers |score => 2 2018-2019 
Modules Peer Review linked to year of linked course via__jof 1 from school PLC with |4 = 26-50% schools with 
Feedback atset  |LEARN tab individual peer review rubric score of 1 Weanie pees 
points in the courses) teacher level _|at least once per 4 (low) = 65-74% of Coe . 
instructional CoreTools data | year per teacher a Or , [2 = 51-75% schools with 
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school level is score 
of 2. 


Teacher 
perception of 
the helpfulness 
of coach written 
feedback on 
modules 


Teacher rating of 
coach-provided 
asynchronous 
feedback via 


CoreTools between 


push-in sessions 


Teacher 


Teacher 
survey 


Spring teacher 
survey 
administered 
by CRESST 
will ask 
teachers to 
report on 
whether they 
found written 
feedback via 
the peer review 
form and 
comments in 
CoreTools 
helpful. 


0 (very low) = not 
helpful 

1 (low) = alittle 
helpful 

2 (moderate) = 
moderately helpful 
3 (high) = very 
helpful 


Adequate 
implementation at 
teacher level is score 
of 2 


School level: 

0 (very low) = less 
than 65% of teachers 
in school PLC with 
score of 2 

1 (low) = 65-74% of 
teachers in school 
PLC with score of 2 
2 (moderate) = 75- 
85% of teachers in 
school PLC with 
score of 2 

3 (high) = over 85% 
of teachers in school 
PLC with score of 2 


Adequate 
implementation at 
school level is score 
of 2. 


Sample-level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 = > 75% schools with 
score 22 


Adequate implementation 
at sample level is a score 
of 2 
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All participating 
schools 


Once per year from 
2016-2017 to 
2018-2019 


3. Key Component = Teacher Implementation Activities 


Module editing | Participating Teacher CoreTools |Data 0 = no evidence of |Adequate School level: Sample-level All participating | Once per year from 
teachers expected data transmitted by |edits to task on any implementation at 0 (very low)=less |0=<25% schools with | Schools 2017-2018 to 
to edit at least one LDC.org to module teacher level is score |than 65% of teachers |score = 2 2018-2019 
module in each CRESST in2 |1 = edited task on [0f2 in school PLC with —_|4 = 26-50% schools with 
yearof data pulls per Jat least one module score of 2 score > 2 
Implementation: aed 2 = edited task in at 1 (low) = 65% to 2 = 51-75% schools with 
a a. module <75% of teachers in score = 2 
and eitner school PLC with = ‘ 
standards OR text score of 2 ene schools with 
in at least one 2 (moderate) = 75- 
module 85% of teachers in . 
3 = edited task in at school PLC with Adequate implementation 
least one module scorer 2 ' a level is a score 
and either ‘ 0) 
3 (high) = over 85% 
elisa OR text of teachers in school 
in at least one PLC with score of 2 
module, and either 
skills and mini- 
tasks or rubric in at Adequate 
least one module. implementation at 
school level is score 
of 2. 
Module Participating Teacher CoreTools |Data 0 (very low) =no —_ | Adequate 0 (very low) =less — |Sample-level All participating | Once per year from 
implementation | teachers expected data transmitted by | student work implementation at than 65% of teachers |Q = < 25% schools with | Schools 2017-2018 to 
to implement two LDC.org to samples uploaded |teacher level is score jin school PLC with —|score > 2 2018-2019 
modules per year CRESST in 2 |to CoreTools of 2 score of 2 


and upload student 
work based on 
modules 


data pulls per 
year 


1 (low) = student 
work samples for 1 
module uploaded 
2 (moderate) = 
student work 
samples for 2 
modules uploaded 
3 (high) = student 
work samples for 3 


1 (low) = 65-74% of 
teachers in school 
PLC with score of 2 
2 (moderate) = 75- 
85% of teachers in 
school PLC with 
score of 2 


1 = 26-50% schools with 
score 22 

2 = 51-75% schools with 
score 2 2 

3 = > 75% schools with 
score 22 
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or more modules 


3 (high) = over 85% 


Adequate implementation 


coach each month. 


progress calls 

2 (moderate) = 9- 
13 planning and 
progress calls 

3 (high) = 14 or 
more planning and 
progress calls 


2 = 51-75% schools with 
score 22 

3 = > 75% schools with 
score 2 2 


Adequate implementation 
at sample level is a score 


of 3 


uploaded of teachers in school Jat sample level is a score 
PLC with score of 2. jof3 
Adequate 
implementation at 
school level is score 
of 2. 
Allindicators |N/A N/A N/A N/A N/A N/A N/A Sample-level for key All participating | Once per year from 
component as awhole _| schools 2017-2018 to 
‘= adequate 2018-2019 
implementation for each 
indicator 
0 = inadequate 
implementation for one or 
more indicator 
Adequate implementation 
at sample level for key 
component is score of 1. 
Frequency of {At minimum, School PLC 0 (very low) = less | Adequate Sample-level All participating | Once per year from 
Coach/Teacher |Teacher Leaders Reflection than 4 planning and |implementation at 0 =< 25% schools with | Schools 2017-2018 to 
Leader monthly |are expected to form progress calls in | school level is score score 2 2 2018-2019 
meetings have 30 minute the year of 2 4 = 26-50% schools with 
planning and 1 (low) = 4-8 score > 2 
progress call with planning and 
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Administrator |One administrator |School LDC 0 (very low) = Adequate Sample-level All participating | Once per year from 
attendance at _| per school is Attendance participating inno implementation at 0 = < 25% schools with {schools 2017-2018 to 
quarterly in- expected to attend Records Aes school level is score score 2 2 2018-2019 
person LDC’s in-person Ow) = of 2 = 96-519 ‘ 
administrator | administrator participating in 1 Were Soe 
meeting meetings, occurring event anne 
three times during 2 (moderate) = 2 = 51-75% schools with 
the school year. participating in 2 score 2 2 
events 3 => 75% schools with 
3 (high) = score = 2 
participating in 3-4 
events 
Adequate implementation 
at sample level is a score 
of 3 
Teacher Leader | Teacher Leader is | School LDC 0 (very low) = Adequate Sample-level All participating | Once per year from 
attendance at expected to attend Attendance participating inno |implementation at 0 =< 25% schools with |schools 2017-2018 to 
quarterly in- LDC's in-person records a school level is score score 2 2 2018-2019 
person Teacher | administrator ow) = of 2 = BF F 
Leader meetings, occurring participating in 1 aise aeaeeei 
meetings three times during event ae ; 
the school year. 2 (moderate) = 2 = 51-75% schools with 
participating in 2 score 2 2 
events 3 = > 75% schools with 
3 (high) = score = 2 
participating in 3-4 
sli Adequate implementation 
at sample level is a score 
of 3 
Principal Mini- | School leaders Teacher Teacher Data collected |0 (low) = teacher |Adequate 0 (very low) =less — |Sample-level All participating | Once per year from 
task expected to survey each Spring __|reports 0 implementation at than 65% of teachers |Q = < 25% schools with | Schools 2016-2017 to 
Observation observe all LDC viaCRESST observations by —_—| teacher level is 1 in school PLC score 2 2 2018-2019 
teachers survey. school leader 
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implementing at 1 (moderate) = reporting adequate | 1 = 26-50% schools with 


least one mini-task teacher reports 1 implementation score 2 2 
observation by 1 (low) = 65% to 74% |2 = 51-75% schools with 
school leader of teachers in school |score > 2 
2 (high) = teacher PLC reporting 3 = > 75% schools with 
reports 2 adequate score > 2 
observations by implementation 
ons eet 2 (moderate) = 75% | adequate implementation 
3 (very high) = to 99% of teachers in | .4 sample level is a score 
teacher reports 3 or school PLC reporting |o¢3 
more observations adequate 
by school leader implementation 


3 (high) = 100% of 
teachers in school 
PLC reporting 
adequate 
implementation 


Adequate 
implementation at 
school level is score 
of 2. 
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Appendix H: 
Outcome Analysis Methodology 


Analysis Model Specification 


For our outcome analyses, we used a threshold of p < .05 to determine whether there was 
a Statistically significant impact of LDC on ELA achievement. In addition to the LDC treatment 
indicator, a teacher effect for years of experience was included, as well as an aggregate 
indicator measuring the mean baseline performance of each student’s classroom peers. The 
fixed effects also included student characteristics to identify the matched comparison sample of 
students, such as baseline achievement, socioeconomic status, demographics, language 
proficiency, grade level, and participation in special education. 


The three-level MMMC model was used to estimate the impacts of the LDC intervention 
on student learning. The general specification for the middle school level MMMC model is 
shown in the following equation using similar notation proposed by Browne et al. (2001, 
equation 6) and applied in Tranmer et al. (2014, equation 3). 


w; ju? + e; 


' (3) 
Vi = xj + us ijUj 


chool(i) 
j€Teacher (i) 


i = 1,...,n Teacher(i) C (1,...,J) 


(3) 2 (2) 2 
Uschool(i) ~ N(0, Oxi(2))> uo~ N(0, Ouny)s e; ~ NCO, a2) 


In this model y; is the student achievement score response, X;is a vector of the fixed 
covariates and f is the vector of the corresponding fixed effects. School(i) is the school which 


student i attends, thus the term y®) 


School(i) TePresents the random effects for that level of 


classification. Within the term > jereacher(i) wij” , u” 


teachers included in the selected dataset, and w, ; is the weight which sums to 1 for each 


is the set of j random effects for the 


student applied in proportion to the instruction time assigned with each teacher. The following 
presents an example of the full middle school model specification. 


achievement; = By + LDC; * B, + priorELA; * Bz + priorMath, * B, + Female; * 6B, + LEP; 
* Bs +SES; * Bg, + Black; * B; + Hispanic; * Bg * +SPED; * By + Gifted; * B49 
+ HonorsELA, * Bi, + priorClassELA; * Byz + TeachExp; * B13 


+ TeachAttend, * B,4 + ae wi jy” +e; 
j€Teacher (i) 
u® ~ N(0,0%,3)) u? ~ N(0,02.)),€; ~ N(O,02) 
School(i) Mu(3) Pe Mu(2yy vi ie 
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In this model, achievement; is the standardized ELA outcome score for student i; In the 
dosage-dependent model, LDC; is the proportion of core class instruction time taken with an 
LDC teacher (range = 0-1). In the dosage-independent model LDC; is coded as zero for 
comparison students and as 1 for students receiving any level of LDC teacher exposure. 


Female, LEP,, Black;, Hispanic; SPED,, Gif ted;,and SES; are student demographic 
indicators coded 1 if the status is present and 0 if absent; 
HonorsELA; is dummy coded 1 when a student was enrolled in an at least one 


honors ELA course and otherwise as 0, allowing students who did not enroll in any 
honors ELA courses to serve as the reference group; 


priorELA; and priorMath, are standardized student achievement scores from the 
baseline year; 


priorClassELA, is the aggregated mean of the baseline ELA scores for all the core 
class peers of student j; 


TeachExp;, is the aggregated percentage of the years of teaching experience less than 
three for those teachers that student i was exposed to in his/her core classes; 


TeachAttend; is the aggregated mean attendance for those teachers that student / 
was exposed to in his/her core classes; 


B, is the impact of LDC, the treatment; 

B12 is the average difference between Cohort 1 and Cohort 2; 

Bz & B3 are the effects of the baseline score covariates; 

B, -- By9 are the effects of the demographic covariates; 

B11 +» By4 are the effects of the aggregated class level covariates; 
(3) (2) 


Uschool(i)’ uj 


respectively assumed to all have a mean of zero and a variance, 
2 2 2 : 
Ou (2) Fu(3) Te respectively. 


,e; are the error components at the school, teacher, and student level 


Student/Teacher Course Exposure Weighting 


Tables H1 and H2 demonstrate how the process of calculating general MMMC teacher 
weights and LDC treatment weights, for the dosage modeling approach, was conducted 
respectively for elementary and middle school. In elementary school, in the event that a 
student was exposed to more than one teacher, each content area was given equal weight in 
distributing teacher/student exposure. For example, if a student was enrolled for both ELA and 
social studies/history under one teacher, then that teacher was coded as .67 for having 
contributed to two thirds of the students’ core curriculum exposure. If the same student 
enrolled in science with a different teacher than the one who was linked to their course marks 
in ELA and social studies/history, then that science teacher would have been coded as .33 and 
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all other teachers in the sample would have been coded as zero. This would then result in the 
student’s exposure adding to a unity (1). 


Table H1 
Example of Elementary School Student/Teacher Weighting Based on Course Links 


Marking Example student/teacher weighting for use in MMMC Example treatment 
period (weight = marking period/total parking period 3) dosage weight 
One Student enrolled with an intervention teacher: Weight = .333 


weight = (1/3) = .333 


Two Student enrolled with an intervention Teacher: Weight = .333 
weight = (1/3) = .333 

Three Student enrolled with a non-intervention teacher: Weight = .000 
weight = (1/3) = .333 


Total Unity: for every student the student/teacher weights sum to 1 Treatment weight = 
.667 


Note. In the selected samples for these analyses, core content courses within each marking period were 
taught by a single teacher. 


In middle school, students’ exposure to teachers at the course level in the three core 
content areas was coded in the same manner as in the elementary grades based on enrolled 
time preceding the assessment period. A difference in our middle school coding process was 
that we did not force each core content area into equal weighting. Instead, each core content 
area exposure contributed to a core content area total sum that formed the basis from which 
the weights were proportioned. 


Most commonly, a student had equivalent core instruction exposure in each of the three 
content areas (often two terms each). In that scenario, if a student had exposure to three 
different teachers, then each teacher would contribute one third (.33) of the overall core 
curriculum exposure and all other teachers in the sample would be coded as zero. However, in 
seventh grade the students often only had one semester of science. It was also occasionally the 
case that students would take both a core and an additional science course; in these cases, 
both science courses were included in the LDC analysis. 


The weighting in middle school was always distributed as a proportion of the total 
semesters across the three content areas. Therefore, if a student accumulated one science unit 
(one semester), two social studies units (two semesters), and two ELA units (two semesters), 
the base number of units would be five. Using that scenario, the science teacher would 
contribute one fifth (.20) of the overall core curriculum exposure with the social studies and 
science teachers contributing two fifths (.40) each, again resulting in the student’s exposure 
adding to a unity (1). If in this scenario the ELA teacher was an intervention teacher, and if the 
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social studies and science teachers were not intervention teachers, the treatment dosage 
weight would equal 0.40 (see table H2). 


Table H2 
Example of Middle School Student/Teacher Weighting Based on Course Mark Links 


Core content Example student/teacher weighting for use Example treatment 
area Course name — in MMMC (weight = subject units/total units) dosage weight 
ELA English 7A& 7B = Student enrolled two terms of core ELA Weight = .400 


(Grade 7) with an intervention teacher: 
weight = 2/5 = .400 


Social studies Social Studies Student enrolled two terms of core social Weight = .000 
WHG: ANC CIV & | studies (Grade 7) with a nonintervention 
B teacher: weight = 2/5 = .400 

Science Science 7 Student enrolled one terms of core science Weight = .000 


(Grade 7) with a nonintervention teacher: 
Weight = 1/5 = .200 


Total Unity: for every student the student/teacher Treatment weight 
weights sum to 1 = .400 


Calculation of Effect Size 


We calculated student-level effect sizes according to the What Works Clearinghouse 
(WWC) 4.0 criteria. Specifically, for the impact analysis with treatment status as a dichotomous 
variable, we calculated Hedges’ g, the difference in adjusted mean outcomes for the groups 
divided by the unadjusted pooled within-group standard deviation of the outcome measure in 
the sample, for all outcomes. The difference in adjusted mean outcomes is estimated by f, in 
the models we previously defined, as outlined in the WWC standards handbook for computing 
effect sizes in multi-level frameworks. Specifically, 

wy 
g eS 
(n; —1)s? + (n, — 1)s?2 
ni tn, — 2 


where y is £,, which is the coefficient from the MMMC for the intervention effect. 
Note that in our analyses the outcome measure is standardized within the analytical 


sample (M = 0, SD = 1). As a result, we expect that g would likely be quite similar to the f, 
coefficient from the MMMC model in the large samples we plan to collect later in the study. 


Though it is not standard to use g with a continuous treatment effect, as in the case of 
our dosage-dependent treatment measure, we have defined and matched populations n; and 
n, where treated students (n;) could have any positive treatment value < 1, and comparison 
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students (n,) had a treatment value of zero. We could therefore calculate g in the case of our 
dosage-dependent treatment measure, and again expect that it would not differ substantially 
from the f, coefficient. It is crucial, however, to note that g and f, in the dosage-dependent 
models reflect the effect size projected for a student who would receive exposure to 
intervention teachers in all of their core classes. Along with this effect we report average 
dosage received by treated students so that the average treatment effect on treated students 
could be calculated. 
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Appendix I: 


Outcome Analysis Tables 


Table |1 


Before and After Matching Sample Sizes: Cohort 2 Elementary School Analysis for Teachers With 2 Years 


of LDC Implementation 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 15 31 707 474 1,977 39,409 
After school matching 15 31 707 72 309 5,677 
Stage 2 
After student matching 15 31 640 71 212 640 


Table 12 


Before and After Matching Sample Sizes: Cohort 2 Middle School Analysis for Teachers With 2 Years of 


LDC Implementation 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 7 32 2,463 115 2,882 80,214 
After school matching 7 32 2,463 35 908 24,497 
Stage 2 
After student matching 7 32 2,299 33 667 2,299 
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Table 13 


2016-2017 Baseline Characteristics of Treated Students Taught by Cohort 2 Elementary School 
Teachers Participating in LDC in 2 Consecutive Years and Comparison Students After Matching 


Treatment group Comparison group 
Student characteristic (n = 640) (n = 640) 

Race/ethnicity 

Hispanic (%) 76.4 76.4 

Black (%) 6.6 6.6 
Asian (%) 4.7 4.7 

White (%) 10.9 10.9 

Other (%) 1.4 1.4 

Female (%) 49.4 47.8 
Special programs status 

Poverty (%) 57.2 57.2 

English language learner (%) 15.8 15.8 

Special education (%) 6.7 6.7 

Gifted (%) 12.8 8.8 
Student baseline achievement 

Mean baseline year mathematics Z score 0.147 0.158 

Mean baseline year ELA Z score 0.158 0.165 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers 0.137 0.091 

Teacher years of experience (% < 3 years) 11.3 15.1 
Grade level at baseline year 

Grade 3 in 2016-2017 (%) 85.0 85.0 

Grade 4 in 2016-2017 (%) 15.0 15.0 
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Table 14 


2016-2017 Baseline Characteristics of Treated Students Taught by Cohort 2 Middle School Teachers 
Participating in LDC in 2 Consecutive Years and Comparison Students After Matching 


Treatment group 


Comparison group 


Student characteristic (n = 2,299) (n = 2,299) 

Race/ethnicity 

Hispanic (%) 93.4 93.4 

Black (%) 3.1 3.1 

Asian (%) 1.0 0.8 

White (%) 1.8 1.8 

Other (%) 0.7 0.9 
Female (%) 51.3 51.3 
Special programs status 

Poverty (%) 61.2 61.4 

English language learner (%) 14.8 14.8 

Special education (%) 6.0 6.0 

Gifted (%) 12.3 14.4 
Student baseline achievement 

Mean baseline year mathematics Z score 0.037 0.024 

Mean baseline year ELA Z score -0.002 -0.011 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.012 -0.028 

Teacher years of experience (% < 3 years) 17.5 14.4 
Grade level at baseline year 

Grade 4 in 2016-2017 (%) 15.1 15.1 

Grade 5 in 2016-2017 (%) 42.2 42.2 

Grade 6 in 2016-2017 (%) 42.6 42.6 
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Table I5 


Baseline Characteristics of Prior Exposure Subgroups of Treated Students Taught by Cohorts 1 and 2 
Middle School and Matched Comparison Student Groups 


Prior and Comparison 
current group for Current only Comparison 
treatment prior and treatment group for 
group current group current only 
Student characteristic (n =1,161) (n = 1,161) (n = 2,133) (n = 2,133) 
Race/ethnicity 
Hispanic (%) 95.9 95.9 94.0 94.0 
Black (%) 3.0 3.0 2.5 2.5 
Asian (%) 0.3 0.2 1.0 0.8 
White (%) 0.5 0.4 1.8 1.8 
Other (%) 0.3 0.5 0.7 0.9 
Female (%) 51.9 51.9 50.4 50.4 
Special programs status 
Poverty (%) 69.4 69.0 72.8 72.0 
English language learner (%) 18.5 18.5 19.6 19.6 
Special education (%) 7.5 7.4 7.9 9.0 
Gifted (%) 14.8 18.7 10.5 11.2 
Student baseline achievement 
Mean baseline year mathematics Z 0.040 0.025 -0.100 -0.092 
score 
Mean baseline year ELA Z score -0.019 0.038 -0.132 -0.132 
Class and teacher characteristics 
Mean baseline ELA Z score of -0.054 -0.056 -0.127 -0.121 
current peers 
Teacher years of experience (<=3) 18.8 14.3 13.6 12.2 
Grade level at baseline year 
In Grade 4 (%) 1.7 1.7 18.4 18.4 
In Grade 5 (%) 28.9 28.9 57.0 57.0 
In Grade 6 (%) 69.3 69.3 24.6 24.6 
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Table I6 


Baseline Characteristics of Current Year Dosage Subgroups of Treated Students Taught by Cohorts 1 and 
2 Middle School and Matched Comparison Student Groups 


Student characteristic 


High dosage Comparison 


treatment group for 
group high dosage 
(n = 1,014) (n = 1,014) 


Low dosage Comparison 
treatment group for low 
group dosage 


(n = 2,280) (n = 2,280) 


Race/ethnicity 
Hispanic (%) 
Black (%) 
Asian (%) 
White (%) 
Other (%) 
Female (%) 
Special programs status 
Poverty (%) 
English language learner (%) 
Special education (%) 
Gifted (%) 
Student baseline achievement 


Mean baseline year mathematics Z 
score 


Mean baseline year ELA Z score 
Class and teacher characteristics 


Mean baseline ELA Z score of 
current peers 


Teacher years of experience (<=3) 
Grade level at baseline year 

In Grade 4 (%) 

In Grade 5 (%) 

In Grade 6 (%) 


95.6 


49.5 


95.6 


49.5 


94.3 94.3 
2.5 2.5 
0.7 1.0 
1.7 1.7 
0.8 0.5 
51.6 51.6 
68.4 67.5 
16.5 16.5 
6.6 7.5 
11.8 14.5 
-0.028 -0.017 
-0.074 -0.075 
-0.075 -0.086 
14.4 12.4 
10.7 10.7 
41.7 41.7 
47.5 47.5 
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Table 17 


Effect Estimates of Cohorts 1 and 2 Middle School Teachers Participating in LDC in 2 Consecutive Years on 
Smarter Balanced ELA Performance, Dosage-Dependent Prior Year Subgroups Model 


Variables 


Dosage-dependent subgroup model 
coefficient (SD) 


Prior and current treatment (n = 1,161) 


Control for prior and current (n = 1,161) 


Current only treatment (n = 2,133) 


Control for current only (n = 2,133) 


0.143 (0.057)* 
0.041 (0.026) 
0.098 (0.055) 


Reference Group in Model 


Level 1 student characteristics 
Hispanic 

Black 

Poverty 

Female 

English language learner 
Special education 

Gifted 

Honors English 

Teacher experience 

Baseline peer ELA Z score 
Baseline year mathematics Z score 
Baseline year ELA Z score 

In Grade 4 baseline 

In Grade 5 baseline 


In Cohort 2 schools 


-0.202 (0.047)* 
-0.374 (0.064)* 
-0.050 (0.017)* 
0.179 (0.014)* 
-0.133 (0.022)* 
0.035 (0.030) 

0.046 (0.024)* 
0.130 (0.036)* 
-0.032 (0.057) 

0.180 (0.037)* 
0.200 (0.013)* 
0.533 (0.014)* 
0.037 (0.041) 

-0.052 (0.025)* 
0.011 (0.029) 
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Table 18 


Effect Estimates of Cohorts 1 and 2 Middle School Teachers Participating in LDC in 2 Consecutive Years on 
Smarter Balanced ELA Performance, Dosage-Dependent Current Year Dosage Subgroups Model 


Variables 


Dosage-dependent subgroup model 


coefficient (SD) 


LDC teacher treatment high dosage (n = 1014) 


Control for high dosage (n = 1014) 


LDC teacher treatment low dosage (n = 2280) 


Control for low dosage (n = 2280) 


0.174 (0.062)* 
0.032 (0.029) 
0.075 (0.057) 


Reference Group in Model 


Level 1 student characteristics 
Hispanic 

Black 

Poverty 

Female 

English language learner 
Special education 

Gifted 

Honors English 

Teacher experience 

Baseline peer ELA Z score 
Baseline year mathematics Z score 
Baseline year ELA Z score 

In Grade 4 baseline 

In Grade 5 baseline 


In Cohort 2 schools 


-0.202 (0.047)* 
-0.373 (0.064)* 
-0.050 (0.017)* 
0.181 (0.013)* 
-0.131 (0.021)* 
0.034 (0.030) 

0.046 (0.024)* 
0.135 (0.036)* 
-0.030 (0.057) 

0.179 (0.037)* 
0.200 (0.013)* 
0.533 (0.014)* 
0.002 (0.041) 

-0.099 (0.025)* 
0.023 (0.030) 
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